r/PowerShell • • Apr 26 '19

Cheating on Windows Powershell

Post image
297 Upvotes

80 comments sorted by

53

u/ihaxr Apr 26 '19

Just install PowerShell on WSL :)

16

u/IntelligentAsk Apr 26 '19

That's Powershell core. Slightly different interested to see how/if Microsoft closes the gap in support for Windows only services.

26

u/xDind Apr 26 '19

11

u/ihaxr Apr 26 '19

I've been using 6.0 Core for a while now and the only issue I have with it is the name :P. So this will be a welcome change.

14

u/Midnight-Alice Apr 26 '19

I personally completely skipped PowerShell Core, finding it to lack the most appealing aspect of PowerShell - being built into Windows. Once Microsoft dropped development for PowerShell and switched to core I cooled on it quite a bit and was never a big fan about the idea of now having to manage non-native dependencies for my Windows automation (which is generally already difficult and inconsistent as it is WITH native PowerShell tooling).

This is good to hear - splitting core like that was a ridiculous, terrible idea and it sounds like Windows adoption rate suffered for it (duh Microsoft!). At this point it's good to hear that they've seen the obvious light, and they should bring it back to native integration ASAP.

8

u/SeeminglyScience Apr 26 '19

This is good to hear - splitting core like that was a ridiculous, terrible idea and it sounds like Windows adoption rate suffered for it (duh Microsoft!). At this point it's good to hear that they've seen the obvious light, and they should bring it back to native integration ASAP.

Slow Windows adoption was likely expected. They switched to a new .NET runtime and focused on making the cross platform experience better. There wasn't much new for Windows PowerShell users, and there was a decent compatibility gap. They weren't standing around wondering where all the Windows users went, they were waiting on .NET Core 3.0 to bridge the gap.

The only real alternative path they could have gone was to delay going cross platform and Core, and maybe released one more FullCLR version. A version that most folks wouldn't have upgraded to anyway.

1

u/setmehigh Apr 27 '19

Does this mean 7 will be bundled with Windows like powershell already is?

1

u/TheIncorrigible1 Apr 27 '19

No, it'll be an optional feature.

1

u/[deleted] Apr 29 '19

I hope not. I very much enjoyed how obligatory it has been so far.

Every extra step I'd have to take to get machines to the same level, in other words a basic setup - stage 0, is one more reason for my managers to just tell me no, and for my colleagues to not even bother with what I have to say.

The power of default options cannot be overstated. Even educated engineers with decades of experience must be treated like users, and users use what's put in front of them.

1

u/MrMunchkin Apr 26 '19

The only real impact it has is on ASP.NET web apps. It's mostly feature parity AFAIK.

31

u/170lbsApe Apr 26 '19

OK Cool. But could somebody enlighten me how and why this would help me in a Enterprise Administration sense? When most my machines are Windows VM's? With a small assortment of Linux. Not being snarky, just very curious.

9

u/IntelligentAsk Apr 26 '19

If you were using Ansible or were already very familiar with bash. Powershell is still the best tool for Windows admin.

2

u/TheIncorrigible1 Apr 26 '19

Ansible has plenty of Windows integration modules built and has for a long while (also, just use DSC).

2

u/IntelligentAsk Apr 26 '19

Definitely I've found it really helpful in helping to manage mixed environments Linux and Windows. You can call it from WSL which is handy too .

0

u/[deleted] Apr 27 '19

[deleted]

2

u/AspieTechMonkey Apr 27 '19

It's right there in the name. ;)

1

u/BlueArcherX Apr 27 '19

dsc is not advertised or intended to be a complete platform. it exists as a framework of tools native to Windows that can be extended, orchestrated, and manipulated by other tool sets/platforms (chef, puppet, whatever). there are a number of YouTube videos with Jeffery Snover where he describes this scenario.

3

u/lavahot Apr 26 '19

I mean, at that rate doesn't it make sense to install powershell onto your linix machines?

1

u/dogfish182 Apr 27 '19

I did it for some azure stuff. Don’t use it and prefer to use the sdk or the azure-cli mostly.

Well actually I prefer to use AWS but....

2

u/lavahot Apr 27 '19

The sdk? Which sdk? For admin stuff?

1

u/dogfish182 Apr 27 '19

Ah sorry the python sdk use it for most interactions. I don’t need to do things like reboot servers much, for that I guess I would just use the cli

28

u/swinny89 Apr 26 '19

It wouldn't. As far as I am aware, WSL is primarily for developers who want easy access to the standard Linux dev environment. As an administrator, I'm much more excited about putting Powershell on Linux, than putting Linux stuff on Windows.

1

u/nerddtvg Apr 27 '19

I love it for quickly manipulating files and searching logs.

0

u/orby Apr 26 '19

As a developer, that is my primary use case. Actually it's primarily to use openssh instead of the one built into Windows

20

u/swinny89 Apr 26 '19

The ssh built into Windows is OpenSSH as of mid 2018.

3

u/shiftedcloud Apr 27 '19

In my last job, I was managing both Windows and Linux machines. WSL beats the hell out of putty or cygwin because its native Linux.

At my new job, I manage mostly Linux and have a Mac. Which works okay, but I would rather have a Windows machine, and use WSL. At least that way I don't have to deal with versions of shit that are almost, but not quite, the Linux version (e.g. gnu grep).

5

u/ka-splam Apr 27 '19

4.7GB logfile, 55 Million lines, most of them repeated that I wanted to strip out:

PowerShell attempt 1: 12 hours, 12GB RAM, didn't finish.

PowerShell, attempt 2: 45 minutes.

Git Bash attempt 1: 9.5 seconds


PS1

PS C:\> (gc vmware.log) -notmatch 'GuestRpc: Channel \d' | sc vmware2.log

PS2

PS C:\> measure-command { 
    gc .\vmware.log -ReadCount 1000 |%{$_}|
    ?{ $_ -notmatch 'GuestRpc: Channel'} |
    ?{ $_ -notmatch' toolbox-dnd'} | 
    set-content vmware3.log 
}

Days              : 0
Hours             : 0
Minutes           : 44
Seconds           : 45

Git Bash:

$ time cat vmware.log | grep -v 'GuestRpc: Channel' | grep -v 'toolbox-dnd' > vmware4.log

real    0m9.633s

7

u/arthurfm Apr 27 '19

Useful tip #1. Don't use Get-Content with large files. Try 'System.IO.StreamReader' instead. It's massively faster...

https://github.com/lazywinadmin/PowerShell/blob/master/TOOL-Get-LogFast/Get-LogFast.ps1

http://www.yusufozturk.info/windows-powershell/powershell-performance-tips-for-large-text-operations-part-1-reading-files.html

Useful tip #2. RipGrep is even faster than grep and is available for Windows, macOS and Linux.

1

u/ka-splam Apr 28 '19

Try 'System.IO.StreamReader' instead. It's massively faster...

It is, but it's soooo much code..

Useful tip #2. RipGrep is even faster than grep and is available for Windows, macOS and Linux.

Ooh, it's written in Rust so it must be good! :D

1

u/[deleted] Apr 28 '19

You asked how I would do it in another thread. The way with too much code, but also with runspaces to break up the line matching into chucks that operate simultaneously on different threads, which would also increase that amount of code, but decrease the time spent enumerating each line and matching on it.

I would also make everything one test by making your regex be :

Where-Object { $_ -NotMatch "GuestRpc: Channel|toolbox-dnd" }

-1

u/GNUandLinuxBot Apr 28 '19

I'd just like to interject for a moment. What you're referring to as Linux, is in fact, GNU/Linux, or as I've recently taken to calling it, GNU plus Linux. Linux is not an operating system unto itself, but rather another free component of a fully functioning GNU system made useful by the GNU corelibs, shell utilities and vital system components comprising a full OS as defined by POSIX.

Many computer users run a modified version of the GNU system every day, without realizing it. Through a peculiar turn of events, the version of GNU which is widely used today is often called "Linux", and many of its users are not aware that it is basically the GNU system, developed by the GNU Project.

There really is a Linux, and these people are using it, but it is just a part of the system they use. Linux is the kernel: the program in the system that allocates the machine's resources to the other programs that you run. The kernel is an essential part of an operating system, but useless by itself; it can only function in the context of a complete operating system. Linux is normally used in combination with the GNU operating system: the whole system is basically GNU with Linux added, or GNU/Linux. All the so-called "Linux" distributions are really distributions of GNU/Linux.

3

u/ka-splam Apr 28 '19

or as I've recently taken to calling it, GNU plus Linux.

or as I've recently taken to calling it, GNU plus Linux plus PowerShell.

2

u/Lee_Dailey [grin] Apr 28 '19

very bad bot!

4

u/ipv6-dns Apr 27 '19 edited Apr 27 '19

1st command reads all file as String and try to match over it, but in Linux shell your example creates pipes and links them (stdout to stdin) and the reading happens by big chunks. You can do the same in Linux:

s=`cat VMware.log`
echo "$s"|grep -v ...

PS2 is more close to reality, but chunk is very small, so you may to try: a) use bigger chunk size (for example multiple Mb) b) use one "Where-Object" with "-and" instead of 2 calls of "Where-Object" c) use -Like/-NotLike instead of -Match d) may be better will be -CLike/-CNotLike to avoid case-manipulations. Try '>' as in Bash to redirect the output

2

u/ka-splam Apr 28 '19

Good suggestions, I've tried a couple of them;

1st command reads all file as String and try to match over it,

It does, I have plenty of RAM on that system (24GB) so I thought it would be good enough to read a <5GB file, and then avoid pipelining it to filter with the regex engine in one operation. Guess not.

try: a) use bigger chunk size (for example multiple Mb)

I tried 60MB and it wasn't finishing quickly, so I tried 6GB as if to read the whole file in one go, and that took the same 45 minutes.

b) use one "Where-Object" with "-and" instead of 2 calls of "Where-Object"

I haven't tried this; I can't predict whether it would be quicker or not.

use -Like/-NotLike instead of -Match d) may be better will be -CLike/-CNotLike to avoid case-manipulations.

-like and -notlike are implemented with the regex engine behind the scenes, so that should be the same speed. Good idea about CLike/CNotLike, I didn't think of that. Grep is case sensitive by default so that might help.

Try '>' as in Bash to redirect the output

That is not the same in PowerShell, it's a wrapper for out-file; I haven't tried it, but my guess is that it won't help.

1

u/ipv6-dns Apr 29 '19

actually in the case of Linux you does not use the shell for search but 3rd part application, written in C: the "grep". You can use the same approach in Windows with Powershell, but as user of 3 grep-like tool, I can recommend you to use "the platinum searcher" or "the silver searcher" (I prefer the last) but never "grep" because it's a slowest one:

- https://github.com/monochromegane/the_platinum_searcher

- https://github.com/ggreer/the_silver_searcher

PS. Platinum searcher is written in Go and it is naturally cross-platform

1

u/[deleted] Apr 29 '19

Select-String already exists. That's the grep tool

1

u/ipv6-dns Apr 29 '19

the guy reported that its performance is not enough for his task

1

u/[deleted] Apr 29 '19 edited Apr 29 '19

Who? No one here has mentioned Select-String other than me.

1

u/ipv6-dns Apr 29 '19

oh, man, you are right. I am very inattentive ¯_(ツ)_/¯

1

u/ipv6-dns Apr 30 '19

Seems, the best solution is to use silver searcher, it got results with grep vs. ag:

GREP:

cat log.log  0,00s user 0,62s system 10% cpu 6,173 total
grep --color=auto --exclude-dir={.bzr,CVS,.git,.hg,.svn} -v   5,64s user 0,27s system 95% cpu 6,172 total
grep --color=auto --exclude-dir={.bzr,CVS,.git,.hg,.svn} -v 'toolbox-dnd'  0,01s user 0,00s system 0% cpu 6,171 total

AG:

ag --nonumbers -v 'GuestRpc: Channel' log.log  4,98s user 0,16s system 99% cpu 5,147 total
ag --nonumbers -v 'toolbox-dnd'  0,00s user 0,01s system 0% cpu 5,146 total

PowerShell solutions are really very slow, for example:

measure-command { Select-String -SimpleMatch -CaseSensitive -NotMatch 'GuestRpc: channel' -Path log.log|Select-String -SimpleMatch -CaseSensitive -NotMatch 'toolbox-dnd' > z }

Select-String is super slow as well as Get-Content, there are issues for this (eg, https://github.com/PowerShell/PowerShell/issues/7537), seems that -Raw option was added to Select-String, but my current Powershell (6.2) does not have it yet. Also I got out of memory on such call:

gc .\log.log -Raw -ReadCount 1000|Select-String -NotMatch 'GuestRpc: channel'
gc : Exception of type 'System.OutOfMemoryException' was thrown.
At line:1 char:1
+ gc .\log.log -Raw -ReadCount 1000|Select-String -NotMatch 'GuestRpc:  ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (:) [Get-Content], OutOfMemoryException
+ FullyQualifiedErrorId : ProviderContentReadError,Microsoft.PowerShell.Commands.GetContentCommand

File log.log is 1.8Gb.

So, the best solution is to use ag tool.

-5

u/GNUandLinuxBot Apr 27 '19

I'd just like to interject for a moment. What you're referring to as Linux, is in fact, GNU/Linux, or as I've recently taken to calling it, GNU plus Linux. Linux is not an operating system unto itself, but rather another free component of a fully functioning GNU system made useful by the GNU corelibs, shell utilities and vital system components comprising a full OS as defined by POSIX.

Many computer users run a modified version of the GNU system every day, without realizing it. Through a peculiar turn of events, the version of GNU which is widely used today is often called "Linux", and many of its users are not aware that it is basically the GNU system, developed by the GNU Project.

There really is a Linux, and these people are using it, but it is just a part of the system they use. Linux is the kernel: the program in the system that allocates the machine's resources to the other programs that you run. The kernel is an essential part of an operating system, but useless by itself; it can only function in the context of a complete operating system. Linux is normally used in combination with the GNU operating system: the whole system is basically GNU with Linux added, or GNU/Linux. All the so-called "Linux" distributions are really distributions of GNU/Linux.

1

u/SomnambulicSojourner Apr 27 '19

It's an old meme sir, but it checks out.

1

u/Lee_Dailey [grin] Apr 28 '19

bad bot! [frown]

2

u/[deleted] Apr 27 '19

That... Not how I would do that in powershell... 🙄

2

u/ka-splam Apr 28 '19 edited Apr 28 '19

How would you do it?

1

u/[deleted] Apr 29 '19

Select-String -Path vmware.log -Pattern 'GuestRpc: Channel \d' -NotMatch | Out-File vmware2.log

1

u/[deleted] Apr 29 '19 edited Apr 29 '19

Dude, in powershell you're handling entire content, in the bash you're doing specific text selection.

Use Select-String in powershell for that and your speed will increase.

3

u/ihaxr Apr 26 '19

Honestly, not much in your case. The neat thing is it's more performant than a VM and it has native access (and fast!) to the local computer's drive via /mnt/c... so if you have some Linux apps / bash or python scripts that you need to run against some files or folders, you can do it with ease.

0

u/[deleted] Apr 26 '19

[deleted]

5

u/ihaxr Apr 26 '19

Performance. It'll work, but will not perform as well as running it on native Linux or under WSL (see: https://www.reddit.com/r/Python/comments/6qndr6/why_is_python_50_faster_under_windows_subsystem/ )

1

u/1RedOne Apr 26 '19

Why have WSL in the first place?

For developers and cross platform stuff.

-1

u/joerod Apr 26 '19

Also cmdlets are no longer being updated for Windows Powershell. Have a look at expand-archive, it has much more functionality in PS 6.

10

u/doomslayer009 Apr 26 '19

I'm not sure I get it, why would I use WSL? I'm just first hearing about it here and reading the link below I just don't get the advantage.

https://docs.microsoft.com/en-us/windows/wsl/install-win10

3

u/IntelligentAsk Apr 26 '19

It's mostly to encourage developers to use windows. But it's worth looking at from a sysadmin point of view.

9

u/NullReference000 Apr 26 '19

WSL was created to win over Linux users who were very familiar with Bash. It’s probably less useful for a Windows sysadmin

5

u/empty_other Apr 26 '19

So far I have found no use for wsl other than fortune-mod (using wsl to print a random quote everytime i open powershell in windows) Every popular and useful linux app already have pre-compiled windows binaries available. If only MS could make a subsystem for mac or android instead. 🤣

1

u/empty_other Apr 26 '19

But i still think wsl is very cool even if i cant find any use for it. 😎

1

u/network_dude Apr 27 '19

Embrace, assimilate, deprecate.

10

u/kiedtl Apr 26 '19

You heretic

4

u/gangstanthony Apr 26 '19

"We are the arm of the Prophets, Arbiter, and you are the blade. Be silent and swift, and we shall quell this heresy without incident."

1

u/pinotkumarbhai Apr 27 '19

What exactly is wsl ?

Actual Linux instance inside windows ?

0

u/dverbern Apr 28 '19

Yeah me too, Windows Sex Lurker? Wet Saucy Lady? Wild Saharan Latrine?

Why not just write Windows Scripting Language, instead of obscuring things for newbies?

1

u/dverbern Apr 28 '19

Oh geez, is it Windows Subsystem for Linux?

1

u/pinotkumarbhai Apr 28 '19

well i know what WSL stands for. I was asking what it is.

1

u/[deleted] Apr 26 '19

WSL is a must for my job... It's the one thing that's saved me from having to move to a macbook.

1

u/SirWobbyTheFirst Apr 27 '19

Until he is asked to Touch the CP and then if he's like me, he's going to get severely greped the fuck out.

Seriously, who thought touch and CP were a good idea for command names?

0

u/terminal_sarcasm Apr 26 '19

Coming from a macbook, wsl has made developing on a pc marginally bearable.

0

u/neko4 Apr 27 '19

WMI is a technology of Windows management. Shells are just a interface. The problem is PowerShell can't access some of the APIs.

0

u/[deleted] Apr 26 '19

[deleted]

5

u/[deleted] Apr 26 '19

OpenSSH is part of Windows. Unless you mean SSH to an external box, in which case why bother when you have WSL?

1

u/[deleted] Apr 26 '19

I ment the second. I have never used openssh on windows. Only putty and WSL

4

u/[deleted] Apr 26 '19

So Windows 10 includes real OoenSSH as a Windows component. Add Fluent Terminal and it's way better than putty.

0

u/[deleted] Apr 27 '19

[deleted]

4

u/SomnambulicSojourner Apr 27 '19

Yeah, it works great. Just enable the ssh feature in programs and features

1

u/[deleted] Apr 28 '19

Excellent question! Yes, you can run an OpenSSH server on Windows using the bundled OpenSSH. Also posh is pwsh now they've Open Sourced it! 🙂

1

u/StevenRCE0 Apr 27 '19

Can't see any problem using OpenSSH with PowerShell.

1

u/oishishou Apr 27 '19 edited May 05 '19

Doesn't support ed25519.

EDIT: Thankfully, turns out I'm wrong. It works.

3

u/SolidKnight May 04 '19

What do you mean? You can make ED25519 certs with keygen on Windows.

1

u/oishishou May 05 '19 edited May 05 '19

Could've sworn it only supports RSA at the moment.

I haven't checked in a few months, but I definitely saw it in documentation. I'm in mobile and can't check now, but I'm going to go digging in a few hours to check (unless you've got a link). I'd love for that to work, as it's all I currently use, and I've been just defaulting to PuTTy.

EDIT: I'm wrong! Yay! It works!

2

u/SolidKnight May 05 '19

I made one about a week ago on Windows 10 1809 and have been using it to connect to my Linux VMs.

It wasn't/isn't supported in Azure Linux VMs.

ssh-keygen -?
usage: ssh-keygen [-q] [-b bits] [-t dsa | ecdsa | ed25519 | rsa]
                  [-N new_passphrase] [-C comment] [-f output_keyfile]
       ssh-keygen -p [-P old_passphrase] [-N new_passphrase] [-f keyfile]
       ssh-keygen -i [-m key_format] [-f input_keyfile]
       ssh-keygen -e [-m key_format] [-f input_keyfile]
       ssh-keygen -y [-f input_keyfile]
       ssh-keygen -c [-P passphrase] [-C comment] [-f keyfile]
       ssh-keygen -l [-v] [-E fingerprint_hash] [-f input_keyfile]
       ssh-keygen -B [-f input_keyfile]
       ssh-keygen -F hostname [-f known_hosts_file] [-l]
       ssh-keygen -H [-f known_hosts_file]
       ssh-keygen -R hostname [-f known_hosts_file]
       ssh-keygen -r hostname [-f input_keyfile] [-g]
       ssh-keygen -G output_file [-v] [-b bits] [-M memory] [-S start_point]
       ssh-keygen -T output_file -f input_file [-v] [-a rounds] [-J num_lines]
                  [-j start_line] [-K checkpt] [-W generator]
       ssh-keygen -s ca_key -I certificate_identity [-h] [-U]
                  [-D pkcs11_provider] [-n principals] [-O option]
                  [-V validity_interval] [-z serial_number] file ...
       ssh-keygen -L [-f input_keyfile]
       ssh-keygen -A
       ssh-keygen -k -f krl_file [-u] [-s ca_public] [-z version_number]
                  file ...
       ssh-keygen -Q -f krl_file file ...

ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (C:\Users\nobody/.ssh/id_ed25519): demo
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in demo.
Your public key has been saved in demo.pub.
The key fingerprint is:
SHA256:L032Z/LjsNQ2ZIVqn5PSyZr/cBbLBZBECkBiuRc1/E4 nobody@NO-PC
The key's randomart image is:
+--[ED25519 256]--+
|    ooo++  o+.   |
|   .... .o ... . |
|     . . ..   o .|
|    . .   E  . o |
|     .  Soo o o..|
|         =.o B.++|
|        . o * /oo|
|         . . %.* |
|            +o+o.|
+----[SHA256]-----+

1

u/oishishou May 05 '19

Fuck yes, you are right! It works! Thanks!

Sometimes, it's nice to find out you're wrong.

Apparently, it's not impossible to get X-forwarding working, too, but it looks to be hit-or-miss. This is awesome.

Working on both Windows 10 and Windows Server 2019.