Migrate 1 tera data

ℹ️ Support
1

Hi all,
i have and letsay consider 1 tera data from my nextcloud server, now i am migrate to another server.

i am using rsync

rsync -a -P -e "ssh -T -c aes128-ctr -o Compression=no -x" source/ user@x.x.x.x:/destination

it seems that takes more than 2 days not finish it yet
any suggestion from all you
i am open and welcome for any suggestion

thanks all

2

anyone perhaps got other idea?

3

raw copy of 1TB takes 22 hours at constant speed of 100MBit/s https://calculators.io/bandwidth/ Depending on your hardware and bandwidth 2 days sounds not really bad. I would expect big files like archives or media to transfer at maximum speed. small files like text or thumbnails are much slower.

If you have physical access to both hosts you could ship the data by external USB drive - this may happen faster… depending on the kind of your data this could take long time as well and you need to copy the data twice (from original host to the drive and from the drive to destination host). If you got the data replicated to you local client you could upload this to new host from there - again subject to hardware and bandwidth limits.

1 Like
4

Why don’t you use compression? Except for already compressed file formats, this can speed up the transfer: https://unix.stackexchange.com/questions/188737/does-compression-option-z-with-rsync-speed-up-backup

1 Like
5

Glad to hear from you @wwe
thanks buddy
finally after 3,5 days the file transfer finish

for replicate, ehm…i think is possibble but i serve nextcloud services to my client
so is not possible asking them to re-upload, hehehe…seem not quite profesional isn’t :grin::v:

anyway glad to hear somebody help me

6

I’ve been reading around if using compression will takes more time
but correct me if i am wrong

so i dont use compression, the command above is the one that i made it through

from lithuania server goes to 2 places

  1. las vegas
  2. singapore

the data is 900+ gigabyte
finish 3,5 days :grin:

Thanks for your help @tflidd

7

I think this is the speed and calculation

i just wondering
btw nice source buddy

8

It depends on the type of data. If these are all zip archives, then you won’t gain anything, if there are a lot of text files, then yes. In the linked question, they mention an article with tests on mixed data and the increase of the effective bandwidth due to compression.

1 Like
9

aaa i see…understand buddy, so sorry my linux knowledge is limited

thanks alot @tflidd

10

@jtdoank please mark the thread as solved and share and learnings (if you had any) so others can see the solution.

  • you transferred data to 2 destination (at same time? = twice amount of data)
  • both transfers are across the globe - especially Singapore is from my experience slow/unstable for data transfers from Europe
  • your result of 3.5 days gives an average of 25MBit/s for 900GB and avg 50MBit/s for 2*900GB

not such bad, I would say…

11

Yes will do @wwe
thanks for reminding me

  • yups, i transfer the data to 2 destination at the same time with the same data using rsync command above
  • both mean from lithuania to las vegas and singapore

Great if it not to bad
thanks all