180
@pivot_pilot ·

SMB copy drops from 110MB/s to 8MB/s after the first two gigabytes Help

Copying a 40GB folder from my desktop to the server over wired gigabit. It starts at about 110MB/s, holds for maybe 20 seconds, then falls off a cliff to 8 to 12MB/s and stays there for the rest of the transfer. Server is a small box with an eight disk array, 16GB of RAM, no SSD cache. Both machines are on the same switch, cables are new, and iperf3 between them shows a steady 940Mbit both ways. Small file copies are fine. Where do I even start?

7 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @flakey_test · 4mo ago · 2 replies

    If it comes back after the scrub, look at one disk in particular. Run a long SMART test on each and watch the pending sector count. A single failing drive drags the whole array down to its speed and often shows no errors in the pool status until much later.

    138
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @chmod_confused · 4mo ago

      Watching per-disk latency during a transfer is even quicker. One disk sitting at ten times the latency of the others points straight at the culprit.

      62
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @quietpackets · 4mo ago · 3 replies

    iperf3 at line rate rules out the network, and the shape of the curve is the giveaway: fast for exactly as long as the write cache holds, then the speed of the actual disks. Twenty seconds at 110MB/s is roughly 2GB, which is a plausible amount of RAM to be used as a write buffer, and once it is full you are seeing what the array can sustain. 8 to 12MB/s is still terribly low for eight disks though, so the next question is what the array is doing. Check whether a scrub or resilver is running, whether one disk is throwing errors and slowing every write, and what the record or stripe size is relative to your workload.

    172
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @pivot_pilot · 4mo ago

      There is a scrub that started Sunday and it is at 61 percent. That is embarrassing but also a relief.

      74
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @quietpackets · 4mo ago

      Not embarrassing, it is the single most common cause of this exact post. Let it finish, retest, and schedule scrubs for a night you are not copying 40GB folders.

      86
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @epoxy_puddle · 4mo ago

    Test with a single 40GB file rather than a folder of mixed sizes as a control. Lots of small files over SMB is a protocol overhead problem, not a disk problem, and the fix is different.

    96
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @stdlib_stef · 4mo ago

    Also worth checking whether the destination dataset has compression or encryption doing something unexpected, and whether the source is a spinning disk that is itself slowing. Half the time the bottleneck is on the machine nobody suspects.

    104
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report