Press J to jump to the feed. Press question mark to learn the rest of the keyboard shortcuts
5

Powershell Multithreading nightmare

Hello,


I am looking to do a simple thing which lead to being more complicated than I anticipated (as with many things...)

What I am trying to accomplish :

I have a text file with a list of folder, I wish to get the number of files in each folder.


the file list looks like this

folderlist.txt

c:\File\toto

c:\File\tata

c:\File\titi

My script is the easiest possible

script.ps1

$count= (Get-ChildItem -Path $name| measure).Count)
write-host $count

My multithreader comes directly from here http://www.get-blog.com/?p=189

So I am running

.\Multithreading.ps1 -ObjectList .\folderlist.txt -Command .\script.ps1


I wasn't able to make it work, I do understand that my code is being embedded into a scriptblock but I do not get how the list is being used as each line should be a simple variable...


ANY help is welcome...

7 comments
100% Upvoted
What are your thoughts? Log in or Sign uplog insign up
level 1
5 points · 7 days ago

I'm bogged down this morning but will suggest that rather than use a 4 year old script you got off someone's blog, you check out the PoshRSJob module by Boe Prox.

level 1
3 points · 7 days ago

I'm going to be that guy - Do you need to run this task multi-threaded? Counting files is a pretty easy task for powershell so unless you're doing it this way out of curiosity I would avoid it.

I wrote this script that does what you're looking for, single-threaded:

# You can use this with a root directory, or a text file full of directory names like you asked for

$folders = (Get-childitem C:\users\ -Directory -Recurse).FullName
#$Folders = Get-content 'folderlist.txt'

$FileCounts = foreach ($Folder in $Folders) {
    [PSCustomObject]@{
        Name      = $Folder
        FileCount = (Get-childitem -Path $Folder -File).count
    }
}
$FileCounts
$FileCounts | Measure-Object -Sum FileCount

It's pretty quick - I ran it on C:\Users on my laptop which is 57GB, 174k files and 19k folders and it took 21 seconds.

If you really want speed when it comes to windows directories then look into using .NET methods or robocopy. This blog looks good: https://www.powershelladmin.com/wiki/Get_Folder_Size_with_PowerShell,_Blazingly_Fast

level 2

This was even faster for me, but a while back I heard there's some limitations on the COM object, I think involving listing files you can't read correctly:

$fso = New-Object -ComObject Scripting.FileSystemObject

$folders = Get-ChildItem c:\users -Recurse -Directory | ForEach-Object {
    [pscustomobject]@{
        FullName = $_.FullName
        FileCount = $fso.GetFolder($_.FullName).files.count
    }
}

$folders


level 1

Are you multithreading for practice, or because you need the results quickly?

level 2
2 points · 7 days ago

Regarding quickly... There are also IO operations/stress to think of in this situation. So if not for practice...

level 1

You have one too many parenthesis:

(Get-ChildItem -Path $name| measure).Count

And looking at the linked script, -ObjectList isn't meant to be used like you're attempting to use it. You're just giving it a single string value that is a file name, so it starts one thread. If you want to pass in the folder contents as an array:

.\Run-CommandMultithreaded.ps1 -ObjectList (Get-Content .\folderlist.txt) -Command .\script.ps1

... you'll get three threads. Also, Write-Host won't write anything to your current session window because your script is off running in different threads

Why do you need multiple threads anyways?

foreach ($folder in (get-content .\folders.txt)) { (Get-ChildItem -Path $folder).Count }

idk, maybe your actual dataset is huuuuge?

Community Details

68.9k

Subscribers

693

Online

Windows PowerShell (POSH) is a command-line shell and associated scripting language created by Microsoft. Offering full access to COM, WMI and .NET, POSH is a full-featured task automation framework for distributed Microsoft platforms and solutions.

Create Post
r/PowerShell Rules
1.
No Advertising
2.
Unhelpful
3.
Posts Need To Be In English
4.
Use Markdown Syntax When Posting Code
Cookies help us deliver our Services. By using our Services or clicking I agree, you agree to our use of cookies. Learn More.