Topics

Yahoo Group transfer update #transferupdate

 

Hi All,

I've been working to increase the speed of the import process. As a reminder, pulling the data out of Yahoo (the export process) has been caught up for some time now. This is fortunate because the most important thing is to save the data before Yahoo deletes it/stops working. And we are able to do that.

The bottleneck is importing the data into Groups.io, specifically importing group messages. Sunday evening I was able to roll out some optimizations to the database. This resulted in us being able to import over twice as many messages yesterday (2.4M) as our previous high water mark. I've made an additional change, which I hope will further speed up the process.

Thanks,
Mark

RCardona
 

Mark,

Thank you for your continued efforts to optimize the movement of Yahoo Groups to your platform as quickly as possible!  We appreciate you!  My group really enjoys their new home here at G.IO! 

Robert

On 11/5/19 12:49 PM, Mark Fletcher wrote:
Hi All,

I've been working to increase the speed of the import process. As a reminder, pulling the data out of Yahoo (the export process) has been caught up for some time now. This is fortunate because the most important thing is to save the data before Yahoo deletes it/stops working. And we are able to do that.

The bottleneck is importing the data into Groups.io, specifically importing group messages. Sunday evening I was able to roll out some optimizations to the database. This resulted in us being able to import over twice as many messages yesterday (2.4M) as our previous high water mark. I've made an additional change, which I hope will further speed up the process.

Thanks,
Mark

 

On Tue, Nov 5, 2019 at 09:50 AM, Mark Fletcher wrote:
I was able to roll out some optimizations to the database. This resulted in us being able to import over twice as many messages 
Congrats!
 
--
J

Messages are the sole opinion of the author, especially the fishy ones.
My humanity is bound up in yours, for we can only be human together. - Desmond Tutu
Note on an occupational hazard: I comment frequently in beta based on my desire for the betterment of groups.io as a product. Only a fraction of the time are my comments and suggestions specifically a matter of self-interest. Don't assume the latter when the former may be the case.

 

Thanks Mark, that shows in the fact that I was seeing about 20-25 groups per day dropping out in the list ahead of mine.  Now it is dropping at a faster rate so I will not have a real number until I do my stats at the end of the day. 

Keep up the great work, and thanks for letting us know.

Janis
 

Thanks, Mark!  We have moved over 150 places in the past two days and still moving! Our positions just moved to 198 and 199 a few minutes ago. I was expecting tge tranfer to take another 10 days. Now I have alerted my group members to expect their Welcome message in 2 or 3 days, or the weekend at tge latest!  We are thrilled with the faster speed!  😄😄😄😄

Thank you so much!

Janis
 

We are at 67 and 68 in the queue at 10 am CST  so we have moved forward  over 100 groups since yesterday afternoon.  We now expect to be transferred today or tonight. Very exciting!
It has been slightly slower the past 30 minutes to 1 hour, maybe a huge group being imported now?

Many Thanks for speeding up the process and getting us to the finish line earlier than we had expected before!

Janis

Melissa Daston
 

Appreciate your tremendous efforts to improve the process.  My group went from #80 to done.  I know it's been a big effort and new costs with new servers to host and to process transfers.  

Out of curiosity are you running your own server farm or leasing space to host the groups?  

Melissa

 

On Wed, Nov 6, 2019 at 9:11 AM Melissa Daston <mdaston@...> wrote:
Appreciate your tremendous efforts to improve the process.  My group went from #80 to done.  I know it's been a big effort and new costs with new servers to host and to process transfers.  

We were able to add 3.4M messages yesterday, besting yesterday's record by 1M messages. We are no longer falling further behind with transfers, and are starting to (slowly) catch up. As of now, we're importing groups that were exported on Oct 26 at 8am.

There is a large variability in the number of messages groups have (the number of messages being the biggest factor for how long it takes to import a group). So if you're watching your spot in the queue, it won't decrease at a steady pace. Some groups have a few hundred messages; the largest group we've imported had over 600k messages. We're currently importing 6 groups at once.

 
Out of curiosity are you running your own server farm or leasing space to host the groups?  

All the machine instances are leased. I've racked enough machines in my time to know I never want to do that again. It also gives me flexibility to quickly add or remove machines at the touch of a button.

Thanks,
Mark