31
u/170lbsApe Apr 26 '19
OK Cool. But could somebody enlighten me how and why this would help me in a Enterprise Administration sense? When most my machines are Windows VM's? With a small assortment of Linux. Not being snarky, just very curious.
9
u/IntelligentAsk Apr 26 '19
If you were using Ansible or were already very familiar with bash. Powershell is still the best tool for Windows admin.
2
u/TheIncorrigible1 Apr 26 '19
Ansible has plenty of Windows integration modules built and has for a long while (also, just use DSC).
2
u/IntelligentAsk Apr 26 '19
Definitely I've found it really helpful in helping to manage mixed environments Linux and Windows. You can call it from WSL which is handy too .
0
Apr 27 '19
[deleted]
2
1
u/BlueArcherX Apr 27 '19
dsc is not advertised or intended to be a complete platform. it exists as a framework of tools native to Windows that can be extended, orchestrated, and manipulated by other tool sets/platforms (chef, puppet, whatever). there are a number of YouTube videos with Jeffery Snover where he describes this scenario.
3
u/lavahot Apr 26 '19
I mean, at that rate doesn't it make sense to install powershell onto your linix machines?
1
u/dogfish182 Apr 27 '19
I did it for some azure stuff. Don’t use it and prefer to use the sdk or the azure-cli mostly.
Well actually I prefer to use AWS but....
2
u/lavahot Apr 27 '19
The sdk? Which sdk? For admin stuff?
1
u/dogfish182 Apr 27 '19
Ah sorry the python sdk use it for most interactions. I don’t need to do things like reboot servers much, for that I guess I would just use the cli
28
u/swinny89 Apr 26 '19
It wouldn't. As far as I am aware, WSL is primarily for developers who want easy access to the standard Linux dev environment. As an administrator, I'm much more excited about putting Powershell on Linux, than putting Linux stuff on Windows.
1
0
u/orby Apr 26 '19
As a developer, that is my primary use case. Actually it's primarily to use openssh instead of the one built into Windows
20
3
u/shiftedcloud Apr 27 '19
In my last job, I was managing both Windows and Linux machines. WSL beats the hell out of putty or cygwin because its native Linux.
At my new job, I manage mostly Linux and have a Mac. Which works okay, but I would rather have a Windows machine, and use WSL. At least that way I don't have to deal with versions of shit that are almost, but not quite, the Linux version (e.g. gnu grep).
5
u/ka-splam Apr 27 '19
4.7GB logfile, 55 Million lines, most of them repeated that I wanted to strip out:
PowerShell attempt 1: 12 hours, 12GB RAM, didn't finish.
PowerShell, attempt 2: 45 minutes.
Git Bash attempt 1: 9.5 seconds
PS1
PS C:\> (gc vmware.log) -notmatch 'GuestRpc: Channel \d' | sc vmware2.logPS2
PS C:\> measure-command { gc .\vmware.log -ReadCount 1000 |%{$_}| ?{ $_ -notmatch 'GuestRpc: Channel'} | ?{ $_ -notmatch' toolbox-dnd'} | set-content vmware3.log } Days : 0 Hours : 0 Minutes : 44 Seconds : 45Git Bash:
$ time cat vmware.log | grep -v 'GuestRpc: Channel' | grep -v 'toolbox-dnd' > vmware4.log real 0m9.633s7
u/arthurfm Apr 27 '19
Useful tip #1. Don't use Get-Content with large files. Try 'System.IO.StreamReader' instead. It's massively faster...
https://github.com/lazywinadmin/PowerShell/blob/master/TOOL-Get-LogFast/Get-LogFast.ps1
Useful tip #2. RipGrep is even faster than grep and is available for Windows, macOS and Linux.
1
u/ka-splam Apr 28 '19
Try 'System.IO.StreamReader' instead. It's massively faster...
It is, but it's soooo much code..
Useful tip #2. RipGrep is even faster than grep and is available for Windows, macOS and Linux.
Ooh, it's written in Rust so it must be good! :D
1
Apr 28 '19
You asked how I would do it in another thread. The way with too much code, but also with runspaces to break up the line matching into chucks that operate simultaneously on different threads, which would also increase that amount of code, but decrease the time spent enumerating each line and matching on it.
I would also make everything one test by making your regex be :
Where-Object { $_ -NotMatch "GuestRpc: Channel|toolbox-dnd" }-1
u/GNUandLinuxBot Apr 28 '19
I'd just like to interject for a moment. What you're referring to as Linux, is in fact, GNU/Linux, or as I've recently taken to calling it, GNU plus Linux. Linux is not an operating system unto itself, but rather another free component of a fully functioning GNU system made useful by the GNU corelibs, shell utilities and vital system components comprising a full OS as defined by POSIX.
Many computer users run a modified version of the GNU system every day, without realizing it. Through a peculiar turn of events, the version of GNU which is widely used today is often called "Linux", and many of its users are not aware that it is basically the GNU system, developed by the GNU Project.
There really is a Linux, and these people are using it, but it is just a part of the system they use. Linux is the kernel: the program in the system that allocates the machine's resources to the other programs that you run. The kernel is an essential part of an operating system, but useless by itself; it can only function in the context of a complete operating system. Linux is normally used in combination with the GNU operating system: the whole system is basically GNU with Linux added, or GNU/Linux. All the so-called "Linux" distributions are really distributions of GNU/Linux.
3
u/ka-splam Apr 28 '19
or as I've recently taken to calling it, GNU plus Linux.
or as I've recently taken to calling it, GNU plus Linux plus PowerShell.
2
4
u/ipv6-dns Apr 27 '19 edited Apr 27 '19
1st command reads all file as String and try to match over it, but in Linux shell your example creates pipes and links them (stdout to stdin) and the reading happens by big chunks. You can do the same in Linux:
s=`cat VMware.log` echo "$s"|grep -v ...PS2 is more close to reality, but chunk is very small, so you may to try: a) use bigger chunk size (for example multiple Mb) b) use one "Where-Object" with "-and" instead of 2 calls of "Where-Object" c) use -Like/-NotLike instead of -Match d) may be better will be -CLike/-CNotLike to avoid case-manipulations. Try '>' as in Bash to redirect the output
2
u/ka-splam Apr 28 '19
Good suggestions, I've tried a couple of them;
1st command reads all file as String and try to match over it,
It does, I have plenty of RAM on that system (24GB) so I thought it would be good enough to read a <5GB file, and then avoid pipelining it to filter with the regex engine in one operation. Guess not.
try: a) use bigger chunk size (for example multiple Mb)
I tried 60MB and it wasn't finishing quickly, so I tried 6GB as if to read the whole file in one go, and that took the same 45 minutes.
b) use one "Where-Object" with "-and" instead of 2 calls of "Where-Object"
I haven't tried this; I can't predict whether it would be quicker or not.
use -Like/-NotLike instead of -Match d) may be better will be -CLike/-CNotLike to avoid case-manipulations.
-like and -notlike are implemented with the regex engine behind the scenes, so that should be the same speed. Good idea about CLike/CNotLike, I didn't think of that. Grep is case sensitive by default so that might help.
Try '>' as in Bash to redirect the output
That is not the same in PowerShell, it's a wrapper for
out-file; I haven't tried it, but my guess is that it won't help.1
u/ipv6-dns Apr 29 '19
actually in the case of Linux you does not use the shell for search but 3rd part application, written in C: the "grep". You can use the same approach in Windows with Powershell, but as user of 3 grep-like tool, I can recommend you to use "the platinum searcher" or "the silver searcher" (I prefer the last) but never "grep" because it's a slowest one:
- https://github.com/monochromegane/the_platinum_searcher
- https://github.com/ggreer/the_silver_searcher
PS. Platinum searcher is written in Go and it is naturally cross-platform
1
Apr 29 '19
Select-Stringalready exists. That's the grep tool1
u/ipv6-dns Apr 29 '19
the guy reported that its performance is not enough for his task
1
1
u/ipv6-dns Apr 30 '19
Seems, the best solution is to use silver searcher, it got results with grep vs. ag:
GREP: cat log.log 0,00s user 0,62s system 10% cpu 6,173 total grep --color=auto --exclude-dir={.bzr,CVS,.git,.hg,.svn} -v 5,64s user 0,27s system 95% cpu 6,172 total grep --color=auto --exclude-dir={.bzr,CVS,.git,.hg,.svn} -v 'toolbox-dnd' 0,01s user 0,00s system 0% cpu 6,171 total AG: ag --nonumbers -v 'GuestRpc: Channel' log.log 4,98s user 0,16s system 99% cpu 5,147 total ag --nonumbers -v 'toolbox-dnd' 0,00s user 0,01s system 0% cpu 5,146 totalPowerShell solutions are really very slow, for example:
measure-command { Select-String -SimpleMatch -CaseSensitive -NotMatch 'GuestRpc: channel' -Path log.log|Select-String -SimpleMatch -CaseSensitive -NotMatch 'toolbox-dnd' > z }Select-String is super slow as well as Get-Content, there are issues for this (eg, https://github.com/PowerShell/PowerShell/issues/7537), seems that
-Rawoption was added to Select-String, but my current Powershell (6.2) does not have it yet. Also I got out of memory on such call:gc .\log.log -Raw -ReadCount 1000|Select-String -NotMatch 'GuestRpc: channel' gc : Exception of type 'System.OutOfMemoryException' was thrown. At line:1 char:1 + gc .\log.log -Raw -ReadCount 1000|Select-String -NotMatch 'GuestRpc: ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidOperation: (:) [Get-Content], OutOfMemoryException + FullyQualifiedErrorId : ProviderContentReadError,Microsoft.PowerShell.Commands.GetContentCommandFile
log.logis 1.8Gb.So, the best solution is to use
agtool.-5
u/GNUandLinuxBot Apr 27 '19
I'd just like to interject for a moment. What you're referring to as Linux, is in fact, GNU/Linux, or as I've recently taken to calling it, GNU plus Linux. Linux is not an operating system unto itself, but rather another free component of a fully functioning GNU system made useful by the GNU corelibs, shell utilities and vital system components comprising a full OS as defined by POSIX.
Many computer users run a modified version of the GNU system every day, without realizing it. Through a peculiar turn of events, the version of GNU which is widely used today is often called "Linux", and many of its users are not aware that it is basically the GNU system, developed by the GNU Project.
There really is a Linux, and these people are using it, but it is just a part of the system they use. Linux is the kernel: the program in the system that allocates the machine's resources to the other programs that you run. The kernel is an essential part of an operating system, but useless by itself; it can only function in the context of a complete operating system. Linux is normally used in combination with the GNU operating system: the whole system is basically GNU with Linux added, or GNU/Linux. All the so-called "Linux" distributions are really distributions of GNU/Linux.
1
1
2
Apr 27 '19
That... Not how I would do that in powershell... 🙄
2
u/ka-splam Apr 28 '19 edited Apr 28 '19
How would you do it?
1
Apr 29 '19
Select-String -Path vmware.log -Pattern 'GuestRpc: Channel \d' -NotMatch | Out-File vmware2.log
1
Apr 29 '19 edited Apr 29 '19
Dude, in powershell you're handling entire content, in the bash you're doing specific text selection.
Use
Select-Stringin powershell for that and your speed will increase.3
u/ihaxr Apr 26 '19
Honestly, not much in your case. The neat thing is it's more performant than a VM and it has native access (and fast!) to the local computer's drive via
/mnt/c... so if you have some Linux apps / bash or python scripts that you need to run against some files or folders, you can do it with ease.0
Apr 26 '19
[deleted]
5
u/ihaxr Apr 26 '19
Performance. It'll work, but will not perform as well as running it on native Linux or under WSL (see: https://www.reddit.com/r/Python/comments/6qndr6/why_is_python_50_faster_under_windows_subsystem/ )
1
-1
u/joerod Apr 26 '19
Also cmdlets are no longer being updated for Windows Powershell. Have a look at expand-archive, it has much more functionality in PS 6.
10
u/doomslayer009 Apr 26 '19
I'm not sure I get it, why would I use WSL? I'm just first hearing about it here and reading the link below I just don't get the advantage.
3
u/IntelligentAsk Apr 26 '19
It's mostly to encourage developers to use windows. But it's worth looking at from a sysadmin point of view.
9
u/NullReference000 Apr 26 '19
WSL was created to win over Linux users who were very familiar with Bash. It’s probably less useful for a Windows sysadmin
5
u/empty_other Apr 26 '19
So far I have found no use for wsl other than fortune-mod (using wsl to print a random quote everytime i open powershell in windows) Every popular and useful linux app already have pre-compiled windows binaries available. If only MS could make a subsystem for mac or android instead. 🤣
1
1
10
u/kiedtl Apr 26 '19
You heretic
4
u/gangstanthony Apr 26 '19
"We are the arm of the Prophets, Arbiter, and you are the blade. Be silent and swift, and we shall quell this heresy without incident."
1
u/pinotkumarbhai Apr 27 '19
What exactly is wsl ?
Actual Linux instance inside windows ?
0
u/dverbern Apr 28 '19
Yeah me too, Windows Sex Lurker? Wet Saucy Lady? Wild Saharan Latrine?
Why not just write Windows Scripting Language, instead of obscuring things for newbies?
1
1
1
Apr 26 '19
WSL is a must for my job... It's the one thing that's saved me from having to move to a macbook.
1
u/SirWobbyTheFirst Apr 27 '19
Until he is asked to Touch the CP and then if he's like me, he's going to get severely greped the fuck out.
Seriously, who thought touch and CP were a good idea for command names?
0
u/terminal_sarcasm Apr 26 '19
Coming from a macbook, wsl has made developing on a pc marginally bearable.
0
u/neko4 Apr 27 '19
WMI is a technology of Windows management. Shells are just a interface. The problem is PowerShell can't access some of the APIs.
0
Apr 26 '19
[deleted]
5
Apr 26 '19
OpenSSH is part of Windows. Unless you mean SSH to an external box, in which case why bother when you have WSL?
1
Apr 26 '19
I ment the second. I have never used openssh on windows. Only putty and WSL
4
Apr 26 '19
So Windows 10 includes real OoenSSH as a Windows component. Add Fluent Terminal and it's way better than putty.
0
Apr 27 '19
[deleted]
4
u/SomnambulicSojourner Apr 27 '19
Yeah, it works great. Just enable the ssh feature in programs and features
1
Apr 28 '19
Excellent question! Yes, you can run an OpenSSH server on Windows using the bundled OpenSSH. Also posh is pwsh now they've Open Sourced it! 🙂
1
u/StevenRCE0 Apr 27 '19
Can't see any problem using OpenSSH with PowerShell.
1
u/oishishou Apr 27 '19 edited May 05 '19
Doesn't support ed25519.
EDIT: Thankfully, turns out I'm wrong. It works.
3
u/SolidKnight May 04 '19
What do you mean? You can make ED25519 certs with keygen on Windows.
1
u/oishishou May 05 '19 edited May 05 '19
Could've sworn it only supports RSA at the moment.
I haven't checked in a few months, but I definitely saw it in documentation. I'm in mobile and can't check now, but I'm going to go digging in a few hours to check (unless you've got a link). I'd love for that to work, as it's all I currently use, and I've been just defaulting to PuTTy.
EDIT: I'm wrong! Yay! It works!
2
u/SolidKnight May 05 '19
I made one about a week ago on Windows 10 1809 and have been using it to connect to my Linux VMs.
It wasn't/isn't supported in Azure Linux VMs.
ssh-keygen -? usage: ssh-keygen [-q] [-b bits] [-t dsa | ecdsa | ed25519 | rsa] [-N new_passphrase] [-C comment] [-f output_keyfile] ssh-keygen -p [-P old_passphrase] [-N new_passphrase] [-f keyfile] ssh-keygen -i [-m key_format] [-f input_keyfile] ssh-keygen -e [-m key_format] [-f input_keyfile] ssh-keygen -y [-f input_keyfile] ssh-keygen -c [-P passphrase] [-C comment] [-f keyfile] ssh-keygen -l [-v] [-E fingerprint_hash] [-f input_keyfile] ssh-keygen -B [-f input_keyfile] ssh-keygen -F hostname [-f known_hosts_file] [-l] ssh-keygen -H [-f known_hosts_file] ssh-keygen -R hostname [-f known_hosts_file] ssh-keygen -r hostname [-f input_keyfile] [-g] ssh-keygen -G output_file [-v] [-b bits] [-M memory] [-S start_point] ssh-keygen -T output_file -f input_file [-v] [-a rounds] [-J num_lines] [-j start_line] [-K checkpt] [-W generator] ssh-keygen -s ca_key -I certificate_identity [-h] [-U] [-D pkcs11_provider] [-n principals] [-O option] [-V validity_interval] [-z serial_number] file ... ssh-keygen -L [-f input_keyfile] ssh-keygen -A ssh-keygen -k -f krl_file [-u] [-s ca_public] [-z version_number] file ... ssh-keygen -Q -f krl_file file ... ssh-keygen -t ed25519 Generating public/private ed25519 key pair. Enter file in which to save the key (C:\Users\nobody/.ssh/id_ed25519): demo Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in demo. Your public key has been saved in demo.pub. The key fingerprint is: SHA256:L032Z/LjsNQ2ZIVqn5PSyZr/cBbLBZBECkBiuRc1/E4 nobody@NO-PC The key's randomart image is: +--[ED25519 256]--+ | ooo++ o+. | | .... .o ... . | | . . .. o .| | . . E . o | | . Soo o o..| | =.o B.++| | . o * /oo| | . . %.* | | +o+o.| +----[SHA256]-----+1
u/oishishou May 05 '19
Fuck yes, you are right! It works! Thanks!
Sometimes, it's nice to find out you're wrong.
Apparently, it's not impossible to get X-forwarding working, too, but it looks to be hit-or-miss. This is awesome.
Working on both Windows 10 and Windows Server 2019.
53
u/ihaxr Apr 26 '19
Just install PowerShell on WSL :)