Comment on Introducing reitti: a selfhosted alternative to Google Timeline
ada@piefed.blahaj.zone 3 weeks agoIt's a 1gig json file that has about 10 years of data. I get multiple repeats of the rabbit timeout in the logs. The Job Status section tells me that it's got just under 9 hours of processing remaining for just over 16,000 in the stay-detection-queue. The numbers change slightly, so something is happening, but it's been going for over 12 hours now, and the time remaining is slowly going up, not down.
reitti-1 | 2025-07-04T03:06:08.820Z INFO 1 --- [ntContainer#2-1] c.d.r.s.p.VisitDetectionService : Detected 61806 stay points for user ada
reitti-1 | 2025-07-04T03:06:17.848Z WARN 1 --- [ntContainer#2-1] o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it
reitti-1 |
reitti-1 | com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 9 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more, class-id=0, method-id=0)
reitti-1 | at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.checkShutdown(BlockingQueueConsumer.java:493) ~[spring-rabbit-3.2.5.jar!/:3.2.5]
reitti-1 | at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.nextMessage(BlockingQueueConsumer.java:554) ~[spring-rabbit-3.2.5.jar!/:3.2.5]
reitti-1 | at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:1046) ~[spring-rabbit-3.2.5.jar!/:3.2.5]
reitti-1 | at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:1021) ~[spring-rabbit-3.2.5.jar!/:3.2.5]
reitti-1 | at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.mainLoop(SimpleMessageListenerContainer.java:1423) ~[spring-rabbit-3.2.5.jar!/:3.2.5]
reitti-1 | at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1324) ~[spring-rabbit-3.2.5.jar!/:3.2.5]
reitti-1 | at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
reitti-1 | Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - delivery acknowledgement on channel 9 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more, class-id=0, method-id=0)
reitti-1 | at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:528) ~[amqp-client-5.25.0.jar!/:5.25.0]
reitti-1 | at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:349) ~[amqp-client-5.25.0.jar!/:5.25.0]
reitti-1 | at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:193) ~[amqp-client-5.25.0.jar!/:5.25.0]
reitti-1 | at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:125) ~[amqp-client-5.25.0.jar!/:5.25.0]
reitti-1 | at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:761) ~[amqp-client-5.25.0.jar!/:5.25.0]
reitti-1 | at com.rabbitmq.client.impl.AMQConnection.access$400(AMQConnection.java:48) ~[amqp-client-5.25.0.jar!/:5.25.0]
reitti-1 | at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:688) ~[amqp-client-5.25.0.jar!/:5.25.0]
reitti-1 | ... 1 common frames omitted
danielgraf@discuss.tchncs.de 3 weeks ago
Thanks for the information. I will try to recreate it locally. In my testing I used a 600MB file and this took maybe 2 hours to process on my server. It is one of these ryzen 7 5825U. Since Reitti tries to do these analysis on multiple cores we start it with 4 to 16 Threads when processing. But the stay detection breaks when doing it that way, so it is locking per user to handle that. If now one of them takes a long time the others will break eventually. They will get resheduled 3 times until rabbitmq gives up.
On what type of system do you run it?
I will add some switches so it is configurable how many threads are opened and add some log statements to print out the duration it took for a single step.
ada@piefed.blahaj.zone 3 weeks ago
i7-8700 with 64GB of RAM
danielgraf@discuss.tchncs.de 3 weeks ago
Hmm, I had hoped you say something like a Raspberry PI :D
But this should be enough to have it processed in a reasonable time. What I do not understand in the moment is that the filesize should not affect it in any way. When importing it 100 Geopoints are bundled, send to RabbitMQ. From there we retrieve them, do some filtering and save them in the database. Then actually nothing happens anymore until the next processing run is triggered.
But this than works with the PostGis DB and not with the file anymore. So the culprit should be there somewhere. I will try to insert some fake data into mine and see how long it takes if i double my location points.
ada@piefed.blahaj.zone 3 weeks ago
I was also trying to set up GPSLogger whilst it was crunching through the backlog, and I manually transferred a file from that app before I had autologging configured. Not sure if that could have done it?
The times don't overlap, as the takeout file is only up until 2023