After deploying the .ovf and spinning up the VMs we noticed that the service-va is running at near 100% cpu usage. I can see the elasticsearch process is having trouble with indexing or something.
Everything feels kindof sluggish to the point I don't want to demo this to anyone in current state. I've tried redeploying twice just to end up with the same elasticsearch index error.
What am I missing here?
From my /var/log/elasticsearch/horizon.log:
[2013-03-19 17:44:17,932][INFO ][node ] [Gravity] {0.19.12}[4915]: stopped
[2013-03-19 17:44:17,932][INFO ][node ] [Gravity] {0.19.12}[4915]: closing ...
[2013-03-19 17:44:17,938][INFO ][node ] [Gravity] {0.19.12}[4915]: closed
[2013-03-19 17:54:37,985][INFO ][node ] [Wingfoot, Wyatt] {0.19.12}[5290]: initializing ...
[2013-03-19 17:54:37,988][INFO ][plugins ] [Wingfoot, Wyatt] loaded [], sites []
[2013-03-19 17:54:39,409][INFO ][node ] [Wingfoot, Wyatt] {0.19.12}[5290]: initialized
[2013-03-19 17:54:39,409][INFO ][node ] [Wingfoot, Wyatt] {0.19.12}[5290]: starting ...
[2013-03-19 17:54:39,520][INFO ][transport ] [Wingfoot, Wyatt] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/10.0.90.231:9300]}
[2013-03-19 17:54:42,529][INFO ][cluster.service ] [Wingfoot, Wyatt] new_master [Wingfoot, Wyatt][LlfyHBamQgagg_X7ksLS8A][inet[/10.0.90.231:9300]], reason: zen-disco-join (elected_as_master)
[2013-03-19 17:54:42,559][INFO ][discovery ] [Wingfoot, Wyatt] horizon/LlfyHBamQgagg_X7ksLS8A
[2013-03-19 17:54:42,610][INFO ][http ] [Wingfoot, Wyatt] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.0.90.231:9200]}
[2013-03-19 17:54:42,610][INFO ][node ] [Wingfoot, Wyatt] {0.19.12}[5290]: started
[2013-03-19 17:54:43,158][INFO ][gateway ] [Wingfoot, Wyatt] recovered [1] indices into cluster_state
[2013-03-19 17:55:38,727][DEBUG][action.index ] [Wingfoot, Wyatt] [audit][0], node[LlfyHBamQgagg_X7ksLS8A], [P], s[STARTED]: Failed to execute [index {[audit][Audit][f04c1bd6-b29b-47a3-ba57-7704ea8be3cb], source[{"baseType":"Audit","objectName":"lastSuccessful
SyncProvisioningCalculatorStats","objectType":"GlobalConfigParameters","values":{"encryptedData":"ENCRYPTED","strKey":"lastSuccessfulSyncProvisioningCalculatorStats","strData":"{\n \"lastSuccessfulSyncStartTime\" : 1363363877348,\n \"lastSuccessfulSyncEndTime\" : 136336
3878084,\n \"lastSuccessfulSyncTotalSyncedObjects\" : 0,\n \"lastSuccessfulSyncPerformedBy\" : \"service-va.adellgroup.net\"\n}"},"actorId":null,"actorUserName":null,"clientId":null,"deviceId":null,"sourceIp":null,"objectAction":"CREATE","recordType":"GlobalConfigParame
ters","recordAction":"CREATE","oldValues":{},"objectId":null,"linkedObjectType":"","linkedObjectId":null,"linkedObjectName":null,"timestamp":1363360278510,"uuid":"f04c1bd6-b29b-47a3-ba57-7704ea8be3cb","organizationId":null}]}]
org.elasticsearch.index.mapper.MapperParsingException: Failed to parse [values.strData]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:324)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:598)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:459)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:507)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:449)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:494)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:438)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:308)
at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:202)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [{
"lastSuccessfulSyncStartTime" : 1363363877348,
"lastSuccessfulSyncEndTime" : 1363363878084,
"lastSuccessfulSyncTotalSyncedObjects" : 0,
"lastSuccessfulSyncPerformedBy" : "service-va.testtest.net"
}], tried both date format [dateOptionalTime], and timestamp number
at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:428)
at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:357)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:181)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:311)
... 13 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "{
"lastSuccessfulSyncStartTime..."
at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:747)
at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:422)
... 16 more
T