Login/Register
Getting Data-Import-Handler to work with Solr 8.10.1
Asked Answered
solrdataimporthandler
P

1

0

I'm using regular Solr 8.10.1 (no Solr Cloud) I created a new core by cd into solr bin directory and ran solr create -c "mytest"

I start it like C:\solr-8.10.1\bin\solr start -p 8983

My folder structure:

solrconfig.xml

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
  <lst name="defaults">
    <str name="config">sample-data-config.xml</str>
  </lst>
</requestHandler>
    
<lib dir="./lib" />

I also tried this, but I get the same error

<lib dir="./solr/lib" />

I copied these files into the server\lib folder: solr-dataimporthandler-8.10.1.jar and solr-dataimporthandler-extras-8.10.1.jar.

- solr-8.10.1
    - server
        - lib
        solr-dataimporthandler-8.10.1.jar
        solr-dataimporthandler-extras-8.10.1.jar
        - solr
            - configsets
                - sample_techproducts_configs
                    - conf
            - mytest
                - conf
                    - lang
                  data-config.xml
                  managed-schema
                  protwords.txt
                  solrconfig.xml A
                  stopwords.txt
                  synonyms.txt
                - data
        solr.xml

sample-data-config

<dataConfig>
<dataSource driver="com.microsoft.sqlserver.jdbc.SQLServerDriver" url="jdbc:sqlserver://localhost:1433;databaseName=test" user="<username>" password="<pwd>" />
    <document name="catalogitems">
        <entity pk="id" name="catalogitem" query="SELECT id,photo from products">
            <field name="id" column="ID" />
            <field name="photo" column="photo" />
        </entity>
    </document>
</dataConfig>

When I run http://localhost:8983/solr/mytest/dataimport?command=full-import

I get error:

HTTP ERROR 500 java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware
URI:    /solr/mytest/dataimport
STATUS: 500
MESSAGE:    java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware
SERVLET:    default
CAUSED BY:  java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware
CAUSED BY:  java.lang.ClassNotFoundException: org.apache.solr.util.plugin.SolrCoreAware
Caused by:
java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(Unknown Source)
    at java.security.SecureClassLoader.defineClass(Unknown Source)
    at java.net.URLClassLoader.defineClass(Unknown Source)
    at java.net.URLClassLoader.access$100(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:538)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Unknown Source)
    at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:538)
    at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:483)
    at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:840)
    at org.apache.solr.core.PluginBag.createPlugin(PluginBag.java:154)
    at org.apache.solr.core.PluginBag.init(PluginBag.java:300)
    at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:130)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1015)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:921)
    at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1448)
    at org.apache.solr.core.CoreContainer.lambda$load$11(CoreContainer.java:869)
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.ClassNotFoundException: org.apache.solr.util.plugin.SolrCoreAware
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    ... 35 more

I also checked the error log via the admin console, where I see:

org.apache.solr.core.SolrCoreInitializationException: SolrCore 'mytest' is not available due to init failure: org/apache/solr/util/plugin/SolrCoreAware
    at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1964)
    at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1937)
    at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)
    at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:525)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:427)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:357)
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
    at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:516)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.solr.common.SolrException: org/apache/solr/util/plugin/SolrCoreAware
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1086)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:921)
    at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1448)
    at org.apache.solr.core.CoreContainer.lambda$load$11(CoreContainer.java:869)
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:218)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ... 1 more
Caused by: java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(Unknown Source)
    at java.security.SecureClassLoader.defineClass(Unknown Source)
    at java.net.URLClassLoader.defineClass(Unknown Source)
    at java.net.URLClassLoader.access$100(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.net.URLClassLoader$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:538)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.net.FactoryURLClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Unknown Source)
    at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:538)
    at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:483)
    at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:840)
    at org.apache.solr.core.PluginBag.createPlugin(PluginBag.java:154)
    at org.apache.solr.core.PluginBag.init(PluginBag.java:300)
    at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:130)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1015)
    ... 9 more
Caused by: java.lang.ClassNotFoundException: org.apache.solr.util.plugin.SolrCoreAware
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    ... 35 more

I already checked here, but I think my lib folder structure is correct?

UPDATE 1

When I have in solr.config: <lib dir="./lib" /> my admin UI loads.

Based on Eric's comments I checked https://solr.apache.org/guide/8_10/libs.html#lib-directories so I replaced the above part in solr.config with

<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />

I also tried:

<lib dir="./lib" />
<lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />

In both latter 2 configurations I still get error HTTP ERROR 500 java.lang.NoClassDefFoundError: org/apache/solr/util/plugin/SolrCoreAware

UPDATE 2

I've now placed solr-dataimporthandler-8.10.1.jar and solr-dataimporthandler-extras-8.10.1.jar in multiple folders:

  1. C:\Databases\solr-8.10.1\dist
  2. C:\Databases\solr-8.10.1\server\lib
  3. C:\Databases\solr-8.10.1\server\solr\mytest\lib

In my C:\Databases\solr-8.10.1\server\solr\mytest\conf\solrconfig.xml

I tried:

  • <lib dir="./lib" />
  • <lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />
  • <lib dir="./server/lib" />
  • <lib dir="${solr.install.dir:../../../..}/server/lib" />

But I keep getting the same error.

Also, I need this jar for multiple cores, so I think I should use folder 1 or 2 above?

Folder structure

enter image description here

UPDATE 3

I have not made any changes to any files/configurations, I just unzipped the download and copied the dataimporthandler jars to some more folders (see my update 2 above).

I see no file solrcore.properties anywhere in my Solr directory (can't find anything on Google either) so I'm not sure where to edit that.

On http://localhost:8983/solr I see:

-DSTOP.KEY=solrrocks
-DSTOP.PORT=7983
-Djava.io.tmpdir=C:\Databases\solr-8.10.1\server\tmp
-Djetty.home=C:\Databases\solr-8.10.1\server
-Djetty.host=0.0.0.0
-Djetty.port=8983
-Dlog4j.configurationFile=C:\Databases\solr-8.10.1\server\resources\log4j2.xml
-Dsolr.default.confdir=C:\Databases\solr-8.10.1\server\solr\configsets\_default\conf
-Dsolr.install.dir=C:\Databases\solr-8.10.1
-Dsolr.jetty.inetaccess.excludes=
-Dsolr.jetty.inetaccess.includes=
-Dsolr.log.dir=C:\Databases\solr-8.10.1\server\logs
-Dsolr.log.muteconsole
-Dsolr.solr.home=C:\Databases\solr-8.10.1\server\solr
-Duser.timezone=UTC
-XX:+AlwaysPreTouch
-XX:+ExplicitGCInvokesConcurrent
-XX:+ParallelRefProcEnabled
-XX:+PerfDisableSharedMem
-XX:+UseG1GC
-XX:+UseLargePages
-XX:-OmitStackTraceInFastThrow
-XX:MaxGCPauseMillis=250
-Xms512m
-Xmx512m
-Xss256k

C:\Databases\solr-8.10.1\server\solr\mytest\core.properties

#Written by CorePropertiesLocator
#Tue Nov 02 14:19:06 UTC 2021
name=mytest

C:\Databases\solr-8.10.1\server\solr\mytest\conf\solrconfig.xml

<lib dir="${solr.install.dir:../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />

But still the same error.

Propjet answered 2/11, 2021 at 16:45 Comment(11)
No it's not. See lib-directories. – Ponderous
Thanks. I checked and added update 1. Could you please check what I'm doing wrong? – Propjet
In solrconfig, directories are resolved as relative to the Solr core’s instanceDir, specifying ${solr.install.dir:../../../..}/dist/ means you have put your jars at that path relative to instanceDir, but it's not the case. The problem is your folder structure. If you need DIH only for that core, setting <lib dir="./lib" /> is ok (actually Solr loads it by default given that a lib/ directory exists), and you just need to move your lib/ folder into the core's directory for auto-loading, eg. solr-8.10.1/server/solr/mytest/lib/some.jar. – Ponderous
I honestly don't know what I'm missing, I've added the jars to a bunch of folders to see if any of them get picked up, but none of them seem to be detected as I keep getting the 500 error. – Propjet
I checked out solr-8.10.1, I see that the DIH plugin is already in the dist/ folder. If you start from the example-DIH config (copied into your core's conf), where the lib directives are already defined, then setting the variable solr.install.dir in solrcore.poperties should work. If you don't want to hardcode or maintain solrcore.poperties, then you should update the fallback path (after the :) according to where is your instance dir relative to your install dir. Which means where you have ../../../.. you should have ../../... – Ponderous
In solrcore.poperties : solr.install.dir=/path/to/solr-8.10.1, and in solrconfig.xml : <lib dir="${solr.install.dir:../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" />. If you left the content of the dist/ folder untouched this should work. – Ponderous
Ok, please see my update 3. I did not yet do anything with this comment as I'm not sure what you mean "If you start from the example-DIH config (copied into your core's conf)". Do I need to copy some folder from "example-DIH" folder to "server" folder? And indeed I prefer not to hardcode or maintain solrcore.properties. – Propjet
The first thing is that you should have the jars you need in solr-8.10.1/dist/. Now, there is a solrconfig.xml in solr-8.10.1/example/example-DIH/solr/solr/conf/, have a look at it and copy the lib directives you need into your core's solrconfig.xml, replacing every ../../../.. by ../../... For solrcore.properties, sorry I meant core.properties. – Ponderous
Alright, that fixed it, thanks! – Propjet
Sorry, 1 last question. I now have to add <lib dir="${solr.install.dir:../../../..}/dist/" regex="solr-dataimporthandler-.*\.jar" /> to each core's solrconfig.xml. Is there a way to do this more centrally so it applies to all cores? – Propjet
Each core has it's own solrconfig.xml, but it's possible to load an external xml file within that config. So let's say you define all the DIH things which are common to all of your cores in one single external file, then you can just load it from solrconfig.xml. If there is no post about how to do that on SO, then ask the question. – Ponderous
D
0

As of August 2022.

DIH team stopped support at Solr 8.9. 8.10 is therefore not supported (yet?).

Link to the repository: https://github.com/rohitbemax/dataimporthandler

Link to a discussion about supporting it in Solr 9: https://github.com/rohitbemax/dataimporthandler/issues/32

Link to a PR to support v9: https://github.com/rohitbemax/dataimporthandler/pull/33

With these known issues:

  • Only MariaDB connector supported right now. The connectors to be used need to be shipped as part of this package, and currently only MariaDB connector is shipped. (No SQLite for instance).
  • Solr Admin UI's Dataimport tab could be a bit glitchy when used with this package
Diaghilev answered 11/8, 2022 at 11:21 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.