Upgrade to PowerShell V2

After I had cursed the command line completion of PowerShell for being so utterly powerless, PowerShell developer Lee Holmes commented:

Thanks for the feedback. Much of this behavior has been improved in the latest CTPs of PowerShell V2.

And he pointed me to CTP3 of PowerShell V2. Well, I thought, if these things really have been improved, then I'm most certainly going to try it out.

I downloaded the CTP3 ("Community Technology Preview 3" – this is Microsoft saying "Alpha"), and was going to install it and see, what has changed. But you know, it couldn't be that easy... even with Microsoft developer pointing you in the right direction... it's still Windows...

Installing and Uninstalling

The download page contained a small paragraph, hidden between the others, instructing me to uninstall the previous version of PowerShell first - I chose to ignore it, thinking it was just another measure to help save Microsoft his ass when something goes wrong.

I started up the installer, clicked next and next... everything was going fine. Until I was confronted with this:

Alright, I thought, if that's really what you want, then I uninstall it.

• I opened up Add/Remove Programs to uninstall it and... it wasn't there. I scanned the list for quite some time, wondering if it's named "PowerShell", "Windows PowerShell", "Microsoft Windows PowerShell" or even something else. Nope... not in that list.
• I went to the Start menu entry of PowerShell... no uninstall option there.
• I went to c:\Program Files\PowerSh... nothing like that over there. Where is the damn thing!?
• Finally found it from c:\Windows\System32\WindowsPowerShell\v1.0. I removed these files, restarted installer, and was greeted with the already familiar error message.
• I tried several more things, which didn't helped either.
• I even searched around the registry, but didn't felt comfortable enough to modify it.

Finally I found uninstall instructions, hidden away inside one Windows PowerShell Blog post.

It appeared, that PowerShell was actually installed not as a separate program, but as a Windows update. And in the Add/Remove Programs dialog I had to check the "show updates" option to be able to see and uninstall it.

After that the installation of PowerShell V2 worked out just fine.

A Miracle Has Happened

It appears that the two things I complained about in the Powerless Completion of PowerShell have indeed been fixed.

Completion now also works with executable files in my $PATH and also with aliases. Additionally to complete cmdlet name like Join-Path I can just type jo and press TAB, instead of being only able to complete the part after the dash. Although I would prefer fir to be completed to firefox not firefox.exe, I'm quite satisfied.

On the filename completion side cd foo\bar is no more expanded to cd 'C:\Documents and Settings\Rene\Desktop\foo\bar'. Although it is still expanded a bit, to .\foo\bar, but this is still a big win in brevity.

So yes, a miracle really has happened. First time in this blog a truly happy end.

Smoking PowerShell Pipes

Everybody likes pipes. Starting from 1972 on when Douglas McIlroy introduced the concept, pipes have been an unreplacable tool for hooking programs together. Pipes just work. Well... at least in UNIX. But you know, this is a blog about me being stuck in Windows.

Real Pipes in UNIX

The main idea of a pipe is that you take the output of one program and connect it to the input of another program. The pipe itself does nothing more than just forwards data. When the output of one program does not fit into the input of another program, then you can pipe it first through a filter program, that transforms the data as needed. But the pipes themselves remain only as a transport layer, carefully carrying data from one program to another and not changing a bit on the way.

The same holds true for real-world pipes. A good pipe is one that doesn't change the aroma on its way from bowl to your mouth. It's the tobacco you want to smoke, not the pipe.

I had a PHP script that generated test data for MySQL database. In UNIX I would have used it as follows:

php create-test-data.php | mysql dbname

I thought that this should also work with PowerShell.

Water Pipes in PowerShell

PowerShell pipes work more like water pipes. The smoke that comes in is sucked through water, changing it's aroma, softening the bitter taste. Water pipes are great, but you shouldn't try to sell them labeled as normal pipes.

When I ran the above code in PowerShell, the data generated by the script wasn't exactly the same that MySQL database received. If it even received it, because the thing crashed along its way.

Actually I didn't even had to pipe it to another program, just redirecting the output to a file changed it considerably:

php create-test-data.php > test-data.sql

Let Me Encode This for You

The first problem was encoding. The output of the script was in UTF-8 encoding. PowerShell wanted to convert the text into his internal UTF-16 representation and then convert it to another encoding when saving to file, because for PowerShell the > operator is equivalent of piping your output to Out-File cmdlet:

php create-test-data.php | Out-File test-data.sql

Luckily Out-File takes -Encoding parameter, which can have the following values: unicode, utf7, utf8, utf32, ascii, bigendianunicode, default and oem. I tried all of them, and the only one that preserved my encoding was oem – which designates single-byte encoding.

Let Me Correct Those Lines

The philosophy of UNIX has been: it's all text. The input of every program is text and the output of every program is text. Except when it's not, and when it's not, then you can't use all the common UNIX text-processing tools on it. Instead you have to use separate tools specific to your binary format. For example you can use ImageMagick to apply all kinds of transformations for images.

This philosophy was recognized by the PowerShell team as one of the great weaknesses of UNIX pipeline. And therefore the mantra of PowerShell has been: it's not all text. It has been the great promise of PowerShell, that it will enable you to work more easily with all kinds of data, not just text.

And this all works out fine, when your program outputs .NET objects. But when it doesn't, the output is treated as text. But it's a water pipe as you remember, so the text isn't just left alone, it's transformed into .NET array, array of lines.

Doesn't look that bad, does it. But the trick is that when the text is split to lines, the line separators are discarded. And when it is put back together, all lines are joined with \r\n. For example this input:

Hello,\n
my name is Rene.\r\n
I'm the author of this blog.\n

...will be converted to this output:

Hello,\r\n
my name is Rene.\r\n
I'm the author of this blog.\r\n

First of all it's dosification - all your nice UNIX file endings will be converted to ugly DOS line endings. Really annoying. Not just annoying – terrible. I think you already know where this is going: binary data.

The SQL generated by my script also included binary data. And you can imagine what this kind of conversion can do to binary data.

There is no built-in mechanism in PowerShell to overcome this problem, although the problem is well known and there exist some ugly workarounds.

Conclusion

At the end of the day PowerShell had failed me. It was the first real job I wanted to do with PowerShell and it failed completely.

If PowerShell really wants to succeed, this behavior has to be corrected. Please, PowerShell, no water inside my pipe.

Thanks to paws22 for sharing the above photo in Flickr under Creative Commons Attribution Noncommercial Share-Alike license.

Powerless Completion in PowerShell

PowerShell has command line completion. It can complete cmdlets, arguments, methods, properties and probably many more things. Which is great. But because this is Windows, there just has to be something terribly wrong with it...

Command Line Completion Essentials

Completion can be provided for a lot of things, but there are two for which I consider completion to be essential:

• commands (programs)
• files and directories

Lets look how the completion of these two essential things is handled in PowerShell.

Program Name Completion

It seems natural, that when we have a command line completion, then we should be able to complete commands. PowerShell doesn't think so. Completion is only provided for the PowerShell-native cmdlets. But simple executable programs are not worthy to receive the mighty PowerShell completion. Completion also doesn't work with aliases.

Actually the completion doesn't even work well with PowerShell cmdlets. For example there is cmdlet Join-Path, which is the only cmdlet starting with letter J. One would think that you just type "J", press TAB and have it completed to "Join-Path". Yeah... but not in PowerShell. Instead you have to type "Join-" and only after that you can press TAB to have it completed.

But as I said in my previous post, in Windows real programs aren't available from command line anyway - so I guess it makes sense to not have completion for these things also.

Filename Completion

But what about filename completion. It did work reasonably well in cmd.exe. It clearly must also work in PowerShell. Well... sort of.

Suppose that you open cmd.exe and want to go to your Desktop directory. You are already in your home directory, so you just type cd and the letter d:

C:\Documents and Settings\Rene>cd d

Then you press TAB and it will be completed to:

C:\Documents and Settings\Rene>cd Desktop

This works the same way in PowerShell. But suppose you want to go two directories with a one step. Suppose you want to go to Desktop\MyDir. So you just type backslash and letter m:

C:\Documents and Settings\Rene>cd Desktop\m

Then you press TAB again and in cmd.exe it will be completed to:

C:\Documents and Settings\Rene>cd Desktop\MyDir

But in PowerShell this will be completed to:

PS C:\Documents and Settings\Rene> cd 'C:\Documents and Settings\Rene\Desktop\MyDir'

PowerShell might call this a feature, but I would certainly call this a regression.

Just Another Day in Windows

PowerShell is nice. In fact it's even quite cool. But like always with Microsoft, they managed to screw up the essentials.

Windows from Command Line

In Linux I used command line every day. It was my filesystem explorer. It was my administration interface. It was my programs lauchpad. It was everything.

But in Windows I haven't really made use of the command line so far. Why so? Why do I spend my time clicking around the filesystem? Why do I hunt programs from Start-menu? Have I lost the spirit of command line?

I think not. Actually I really want to use the command line. But there is something really wrong with the whole command line thing in Windows.

The Default Powerless Shell

First I though that the problem is in power, that the default cmd.exe gives you so few possibilities in combining different programs together. So I installed Windows PowerShell.

PowerShell gives you a neat programming environment where you have access to the whole functionality of .NET framework. The command set is well designed and quite easy to learn with the aid of reflection capabilities it provides. It's a lot more UNIX-like than cmd.exe – it even greets you with the familiar UNIX commands like ls, rm, cp, mv, cat, ps and kill.

So the power problem was now solved, and I spent quite a bit of time playing with my new powerful toy, but I still didn't use command line for everyday tasks like I did in Linux.

Command Line Without Commands

The real problem is that in Windows you don't have any commands to run from command line.

I mean, you don't have any interesting commands to run.

Let me explain...

Of course you have commands. Powershell has a lot of built-in commands. But you can't do a whole lot with these commands alone. You really want to invoke real programs (.exe files) to do some hard work.

Again, executables are also available from command line. There are actually a whole bunch of them available. In my system I have 372 programs available from command line:

accwiz	fc	ntvdm	sort
actmovie	find	odbcad32	spdwnwxp
AEJUninstall40	findstr	odbcconf	spider
ahui	finger	osk	spnpinst
alg	fixmapi	osuninst	spoolsv
append	fltmc	packager	sprestrt
arp	fontview	pathping	spupdsvc
at	forcedos	pentnt	spupdwxp
ati2evxx	freecell	perfmon	stimon
Ati2mdxx	fsquirt	ping	subst
atiadaxx	fsutil	ping6	svchost
atiiprxx	ftp	powercfg	syncapp
atiphexx	gdi	powershell	sysedit
atiprbxx	grpconv	print	syskey
atiptaxx	help	progman	sysocmgr
atmadm	hh	proquota	systray
attrib	hostname	proxycfg	ZyDelReg
auditusr	ie4uinit	putty	taskman
autochk	ieudinit	pxcpya64	taskmgr
autoconv	iexpress	pxcpyi64	tcmsetup
autofmt	imapi	pxhpinst	tcpsvcs
autolfn	ipconfig	pxinsa64	telnet
blastcln	ipsec6	pxinsi64	tftp
bootok	ipv6	qappsrv	tourstart
bootvrfy	ipxroute	qprocess	tracert
cacls	irftp	QuickTimeUpdateHelper	tracert6
calc	IsUninst	qwinsta	tscon
charmap	java	rasautou	tscupgrd
chkdsk	javaw	rasdial	tsdiscon
chkntfs	javaws	rasphone	tskill
cidaemon	jdbgmgr	rcimlby	tsshutdn
cisvc	jview	rcp	tzchange
ckcnv	krnl386	rdpclip	twunk_16
cleanmgr	label	rdsaddin	twunk_32
cliconfg	lights	rdshost	unins000
clipbrd	lnkstub	recover	uninst_Zyxel
clipsrv	locator	redir	unlodctr
clspack	lodctr	reg	unsecapp
cmd	logagent	regedit	upnpcont
cmdl32	logman	regedt32	ups
cmmon32	logoff	regini	user
cmstp	logonui	regsvr32	userinit
comp	lpq	regwiz	usrmlnka
compact	lpr	replace	usrprbda
conime	lsass	reset	usrshuta
control	magnify	rexec	utilman
convert	makecab	route	uwdf
cscript	mdm	routemon	w32tm
csrss	mem	rsh	wbemtest
ctfmon	migpwd	rsm	wdfmgr
dcomcnfg	mmc	rsmsink	verclsid
ddeshare	mmcperf	rsmui	verifier
debug	mnmsrvc	rsvp	wextract
defrag	mobsync	rtcshare	WgaTray
dfrgfat	mofcomp	runas	wiaacmgr
dfrgntfs	mountvol	rundll32	winchat
diantz	mplay32	runonce	WinFXDocObj
diskpart	mpnotify	rwinsta	winhelp
diskperf	mrinfo	savedump	winhlp32
DivXCodecVersionChecker	MRT	sc	winhlp32
dllhost	mscdexnt	scardsvr	winlogon
dllhst3g	msdtc	scrcons	winmgmt
dmadmin	msfeedssync	sdbinst	winmine
dmremote	msg	services	winmsd
doskey	mshearts	sessmgr	WinSCP3
dosx	mshta	setdebug	winspool
dplaysvr	msiexec	sethc	winver
dpnsvr	mspaint	setup	wjview
dpvsetup	msswchx	setupn	wmiadap
drmupgds	mstinit	setver	wmiapsrv
drwatson	mstsc	sfc	wmiprvse
drwtsn32	napstat	shadow	wowdeb
dumprep	narrator	share	wowexec
dvdplay	nbtstat	sherlock2	wpabaln
dvdupgrd	nddeapir	shmgrate	wpdshextautoplay
dwwin	net	shrpubw	wpnpinst
dxdiag	net1	shutdown	write
edlin	netdde	sigverif	wscntfy
esentutl	netsetup	skeys	wscript
eudcedit	netsh	slrundll	vssadmin
eventvwr	netstat	slrundll	vssvc
exe2bin	nlsfunc	slserv	wuauclt
expand	notepad	smbinst	wuauclt1
explorer	notepad	smlogsvc	WudfHost
ExportController	nslookup	smss	wupdmgr
extrac32	ntkrnlpa	sndrec32	xcopy
fastopen	ntoskrnl	sndvol32
faxpatch	ntsd	sol

But these aren't interesting programs. Most of them are used by Windows internally and I have no idea what for, and the ones I know about (like notepad, mspaint, explorer, ping) are of very minimal use.

Most of the programs I actually use every day aren't available from command line. Programs like Opera, Emacs, Skype and Windows Media Player.

In Linux all programs are available from command line. If you aren't on command line – you don't exist. While in Windows the command line seems to be for suckers. Real programs are available from Start-menu.

The Problem With C:\Program Files

All this is not the fault of the programs. It's the architecture of Windows that makes adding command line programs so cumbersome that you really want to avoid it if you can.

The locations for command line programs are defined by the PATH variable. In UNIX the PATH contains directories like /bin, /usr/bin, /usr/local/bin and maybe a few more. When new program is installed the executable is placed into one of these /bin-directories and through this becomes naturally available in command line.

But when a new program is installed on a Windows machine, it is placed inside a folder of its own. opera.exe goes inside C:\Program Files\Opera and firefox.exe inside C:\Program Files\Mozilla Firefox. To make a program available from command line its location needs to be added to the PATH, making it longer and longer and even longer.

I have about 80 folders under C:\Program Files. To make all these programs available from command line, I would have to add 80 new paths to the PATH variable. And the PATH is quite long already.

Imagine a PATH like this:

C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Fil
es\ATI Technologies\ATI Control Panel;C:\ProgramFiles\QuickTime\QTSyst
em\;C:\Program Files\WinSCP3\;C:\Program Files\Smart Projects\IsoBuste
r;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\Program Files\7-Zip;C:
\Program Files\Acro Software;C:\Program Files\Adobe;C:\Program Files\A
ltiris;C:\Program Files\AMD;C:\Program Files\ArcGIS;C:\Program Files\A
TI Technologies;C:\Program Files\AuthenTec;C:\Program Files\Avira;C:\P
rogram Files\Bentley;C:\Program Files\BitTorrent;C:\Program Files\CCle
aner;C:\Program Files\Common Files;C:\Program Files\ComPlus Applicatio
ns;C:\Program Files\CONEXANT;C:\Program Files\DIFX;C:\Program Files\Di
giDoc;C:\Program Files\DivX;C:\Program Files\DNA;C:\Program Files\Easy
Internet signup;C:\Program Files\emacs;C:\Program Files\eMule;C:\Prog
ram Files\Fingerprint Sensor;C:\Program Files\foobar2000;C:\Program Fi
les\GIMP-2.0;C:\Program Files\Google;C:\Program Files\GPLGS;C:\Program
Files\Hewlett-Packard;C:\Program Files\HPQ;C:\Program Files\Ideelabor
;C:\Program Files\Internet Explorer;C:\Program Files\Internet Explorer
Developer Toolbar;C:\Program Files\InterVideo;C:\Program Files\IT Are
ndus;C:\Program Files\Java;C:\Program Files\jEdit;C:\Program Files\Kar
en's Power Tools;C:\Program Files\KONICA MINOLTA;C:\Program Files\Logi
tech;C:\Program Files\Messenger;C:\Program Files\microsoft frontpage;C
:\Program Files\Microsoft Office;C:\Program Files\Microsoft Script Deb
ugger;C:\Program Files\Microsoft Visual Studio;C:\Program Files\Movie
Maker;C:\Program Files\Mozilla Firefox;C:\Program Files\Mozilla Thunde
rbird;C:\Program Files\Mozilla Thunderbird_new;C:\Program Files\mplaye
r;C:\Program Files\MSN;C:\Program Files\MSN Gaming Zone;C:\Program Fil
es\MSN Messenger;C:\Program Files\MSXML 4.0;C:\Program Files\MySQL;C:\
Program Files\NetDrive;C:\Program Files\NetMeeting;C:\Program Files\No
rton AntiVirus;C:\Program Files\Notepad++;C:\Program Files\Omnikey;C:\
Program Files\Online Services;C:\Program Files\Opera;C:\Program Files\
Outlook Express;C:\Program Files\pgAdmin III;C:\Program Files\Power Ar
chitect;C:\Program Files\Program Shortcuts;C:\Program Files\PuTTY;C:\P
rogram Files\Quest Software;C:\Program Files\QuickTime;C:\Program File
s\RegCure;C:\Program Files\Skype;C:\Program Files\Synaptics;C:\Program
Files\TortoiseCVS;C:\Program Files\TortoiseSVN;C:\Program Files\Visua
lSubSync;C:\Program Files\Winamp;C:\Program Files\Windows Media Connec
t;C:\Program Files\Windows Media Connect 2;C:\Program Files\Windows Me
dia Player;C:\Program Files\Windows NT;C:\Program Files\WinLemm;C:\Pro
gram Files\WinMerge;C:\Program Files\WinSCP3;C:\Program Files\xerox;C:
\Program Files\XP Codec Pack;C:\Program Files\ZTE EV-DO;C:\Program Fil
es\ZyXEL Communications Corporation

This is scary...

So I guess this finally explains why you can't make full use of Windows from command line alone.

Installing Windows PowerHell

As every child knows, the command prompt Windows provides is a total piece of crap. cmd.exe is to Bash as MS Paint is to Gimp.

But I had heard that Microsoft has another shell to offer: Windows PowerShell. The name itself sounded promising and so I decided to give it a try. But you know... I was stunk in Windows.

Downloading from Microsoft.com

I read a bit of the Wikipedia entry about PowerShell and from there went to the PowerShell homepage.

I clicked on the download link and was presented with a page explaining How to Download Windows PowerShell 1.0. (Although the text on this page didn't really explain anything and sounded more like some marketing gibberish.) I decided that although I have no idea what RTW means, Windows PowerShell 1.0 RTW is probably the version that I need. I chose x86 and English language Package for Windows XP.

On the next page I was informed that the download requires validation that Microsoft has received money for my Windows installation. They really wanted to make the process painful... after all I was running Windows XP (eXtra Painful).

I downloaded GenuineCheck.exe. Ran it. Copied the generated code to the web page. Submitted it. And was finally able to click the download button.

After I had ignored the advice of installer to close all my programs and back up my system, the PowerShell was in my machine.

We'll See What Happens

So now I'm going to use this shell for a while and we shall see what kind of surprises it will bring to our table.