If the folders contain a lot of subfolders and small files, it will take a longer time. Say for example if you transfer a folder with lots of html pages that contains their associated _files folders, then it will take a lot of time. What you can do here is to compress or zip them in to a single file, then transfer it. This way is very fast.
See if your USB ports are 2.0 . if its 1.1 then the speed will be very slow.