由 dawei
我有一个mysql数据库,在一个表中有超过400,000,000条记录.
当我设置限制时,Solr导入运行正常.我的最后一次测试是10,它在大约5分钟内成功运行.在Solr中拥有完整的数据集是必需的,当我删除限制时,solr报告它已经丢失了与mysql的连接.
我相信我遇到的是当前配置中的错误配置或限制.
这是来自tomcat的catalina.out日志
Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT id as solr_id,id,firstname,lastname,state from names Processing Document # 1
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:621)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:327)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:225)
... 3 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: SELECT id as solr_id,state from names Processing Document # 1
at org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72)
at org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.MySQL日志
130616 16:15:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130616 16:15:29 [Warning] option 'max_allowed_packet': unsigned value 53687091200 adjusted to 1073741824
130616 16:15:29 InnoDB: Initializing buffer pool,size = 22.0G
130616 16:15:30 InnoDB: Completed initialization of buffer pool
130616 16:15:31 InnoDB: Started; log sequence number 49 1885610091
130616 16:15:31 [Note] Event Scheduler: Loaded 0 events
130616 16:15:31 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.69' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
MySQL表
CREATE TABLE `names` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,`state` varchar(12) DEFAULT NULL,`source` varchar(128) DEFAULT NULL,`lastname` varchar(128) DEFAULT NULL,`firstname` varchar(128) DEFAULT NULL,PRIMARY KEY (`id`),KEY `source-state` (`source`,`state`),KEY `first-last` (`firstname`,`lastname`),KEY `first-last-state` (`firstname`,`lastname`,`state`)
) ENGINE=InnoDB AUTO_INCREMENT=433924211 DEFAULT CHARSET=utf8;
MySQL配置
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
wait_timeout=66600
net_read_timeout=66600
interactive_timeout=666666
max_allowed_packet=50G
max_connect_errors=666666
innodb_buffer_pool_size=22GB
innodb_flush_log_at_trx_commit = 2
thread_cache_size=4
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
Solr data-config.xml
最佳答案
您可以创建多个根实体以及LIMIT子句,如 –
直到你需要的行
要么
您可以传递限制和偏移参数作为请求参数,state from names limit ${dataimporter.request.startAt},${dataimporter.request.count}">
第二个选项的查询链接如下所示 – http：//< hostname>：< port> / solr / dataimport？command = full-import& startAt = 0& count = 10000000