IE9 and hidden Flash movies

Internet Explorer 9 (and Chrome sometimes as well) tries to be a little too smart and do too many things for you. The best example is removing the Flash movie from the DOM when you set its style to:

display: none !important;.

This small gotcha led to some headaches about why JS wasn’t able to call methods (via the ExternalInterface) on my Flash movie. As it turned out this way due to IE9 just completely removing the Flash movie from the DOM.

Flash Player Debugger on Ubuntu

After having spent a good part of an hour on trying to install the Flash Player Debugger for FireFox on Ubuntu 11 64-bit, I found the following steps worked:

  1. sudo apt-get install nspluginwrapper
  2. copy libflashplayer.so (from debugger download) to /usr/lib/flashplugin-installer/
  3. sudo nspluginwrapper -i /usr/lib/flashplugin-installer/libflashplayer.so
  4. restart FireFox

Thanks to: AskUbuntu.com.

Don’t forget to set up mm.cfg.

When all is done, run:

tail -f ~/.macromedia/Flash_Player/Logs/flashlog.txt

Recording the microphone

Recording the microphone using AS3 is quite easy. Actually, storing the data or compressing it and sending it to a server is where it gets complicated. Normally, a Google search helps a lot when trying to find good information, but on this particular subject there seems to be a lot of misinformation. There are also a lot of libraries and / or frameworks available, but so far they didn’t really do what I wanted to do or did it in a far too complicated way. I’ve provided a list of audio libraries I tried out at the bottom.

Clearing things up

When you record data from the microphone using SampleDataEvents, the actual data is (part of a) 32-bit floating point (Big Endian) 44.1 kHz mono PCM audio stream (assuming that you left the rate of the Microphone at its default of 44). To store the microphone input, all you need to do is append this data to a buffer.

To play back the microphone input you can use the Sound class. The Sound class can only deal with a 32-bit floating point 44.1 kHz stereo PCM audio stream as input. Lucky for us, we’re almost there, we just need to convert mono into stereo. As you can see in the example below, creating (fake) stereo out of a mono signal is a matter of writing each sample twice.

The documentation for the Sound class shows an example of how to merge several Sounds into one using the extract method. We can apply the same technique to play back our ByteArray using the Sound class:

  1. // The buffer where we stored the microphone input.
  2. var playbackBuffer:ByteArray;
  3. // The number of channels the buffer contains,
  4. // 1 means mono, 2 means stereo.
  5. var channels:int = 1;
  6. var outputSound:Sound = new Sound();
  7.  
  8. outputSound.addEventListener(SampleDataEvent.SAMPLE_DATA,
  9. outputSampleDataHandler);
  10. outputChannel = outputSound.play();
  11.  
  12. private function outputSampleDataHandler(event:SampleDataEvent):void {
  13.     for (var i:int = 0; i < 8192 &&
  14.         playbackBuffer.bytesAvailable > 2; i++) {
  15.  
  16.         var sample:Number = playbackBuffer.readFloat();
  17.         event.data.writeFloat(sample);
  18.  
  19.         // Fake stereo as dual mono when the
  20.         // original data is only 1 channel.
  21.         if(channels > 1) {
  22.             event.data.writeFloat(sample);
  23.         }
  24.     }
  25. }

The above seems simple enough, but when you want to save the data to a file, things get complicated. First you need to choose a file format in which to store the output. Secondly, if you want to reduce the amount of data to be saved, you need to choose a codec to compress the audio stream. When I was experimenting with MP3 encoding of my recorded data, I found out that the codec required the stream input to be a 16-bit signed integer with a sampling rate of 44.1 kHz. When you get into this territory, it helps to read up a bit about the techniques behind digital audio. I’ll try to summarize some of the most useful things I learned.

Pulse-code modulation (PCM)

Pulse-code modulation is a way of representing analog audio signal on a digital system by sampling the signal at a fixed interval. See 1) in the image I created to explain some PCM examples.

By keeping the same playback frequency, but adding more samples (using some form of interpolation), you slow down playback / lower the pitch (blue curve). The opposite is also true, remove samples and playback will speed up / the pitch is higher (red curve). See 2) in the PCM examples image.

If you multiply or divide the each sample by a fixed value, you increase or decrease the signal strength (green curve). See 3) in the PCM examples image. Keep in mind that multiplying or dividing could result in clipping or complete cancellation of the signal if you overflow or underflow the bit depth precision.

Sample rate conversion

The sample rate defines how many samples are taken of the analogue sound per second and is expressed in Hertz, for example, CDs have a sample rate of 44.1 kHz. The higher the sample rate, the better the quality. Other typical sample rates are 8, 11.025, 22.05 and 48 kHz.

There are two types of sample rate conversion, there is upsampling and there is downsampling.

It is usually easier (whether you are upsampling or downsampling) to first upsample (using linear interpolation) to the least common multiple (LCM) and then downsample to the required sample rate.

Generally speaking when you downsample an audio signal you can get away with duplicating a sample when upsampling to the LCM, most of the data will be dropped anyway as you downsample to the required sample rate.

Since sample rate conversion is quite CPU intensive, there are some shortcuts you can take to convert to a certain sample rate from another. Say you want to convert 44.1 kHz into 8 kHz, it seems a bit redundant to first upsample to 3528000 Hz (which is the LCM), and then downsample to 8 kHz. What you could do instead, is alternate between using every 5th and 6th sample (since 44.1 / 8 is roughly 5.5).

Bit depth conversion

The bit depth determines granularity of a single sample, the higher the bit depth, the more accurate the sound can be represented. CDs have a bit depth of 16-bit, other typical bit depths are 8, 24 and 32.

Usually 8-bit audio streams are using unsigned integers, 16-bit ones are using signed integers and 32-bit streams use signed floating point. Knowing this it is quite straightforward to convert from one bit depth to the other, it’s a matter of multiplication.

Note: You may need to do some additional checks to make sure you aren’t overflowing the available bits when doing the conversion.

When you understand the theory I just talked about, it becomes a lot easier to write the tools you need to convert audio from one bit rate to the other as well as changing between sample rates. It also gave me some ideas about some filters to implement, like a normalization filter. To help you get started I’ve included some tips and references below.

Tips

  • The Microphone.rate property accepts values of 44, 22, 11 and 8. These values actually represent 44100, 22050, 11025 and 8000 Hz.
  • The WaveEncoder class of the MicRecorder library comes in quite handy when you want to save your recorded audio. Keep in mind that the encode method expects the passed in data to be 32-bit and will convert it to 16-bit on the fly. This tripped me up a couple of times as I passed in a 16-bit audio stream and was surprised to find out that the saved WAV file just produced a lot of white noise. On my copy of this class, I added an additional check to the create method, to make sure that the 16-bit space wasn’t being overflown when converting and multiplying by a _volume (multiplier) value.
  • When you want to output your recorded audio, make sure to keep the Endianness of your ByteArrays to Little Endian. I wondered countless times why playback of my ByteArray resulted in just noise, only to find out that I had forgotten about the Endianness of it.
  • Forget about the loadCompressedDataFromByteArray and loadPCMFromByteArray methods in the Sound class for Flash Player 11. They just don’t work, you’ll get a sound playing, but the last set of bytes seem to loop forever when the audio stream should be finished.
  • If you want to see how the sound you produced actually looks like, you may want to download sofware like Audacity. This tool can give you a visual representation of the wave form as well as a lot of information about the file that you opened. When you open a raw audio stream, you can also flick the settings around to figure out what sampling rate and bit depth the stream is using.

Further reading

AS3 audio libraries

Alchemy on Windows7

Installing Adobe Alchemy on Windows7 wasn’t as straightforward as the Adobe Labs site made me believe.

The following error would keep on showing when I tried to run the example:

$ gcc stringecho.c -O3 -Wall -swc -o stringecho.swc
llvm-gcc.exe: error while loading shared libraries: ? …

I finally got it to work by using the steps below:

  1. Install the Alchemy Repack, from www.covergraph.com in a temporary directory (I just want to copy some of the files).
  2. Download and install Alchemy from the Adobe website, as per their instructions.
  3. Copy the “achacks”, “avm2-libc”, “bin” and “flashlibs” folders from your “Alchemy Repack” installation folder.
  4. Uninstall / Delete Alchemy Repack
  5. Good to go! 🙂
  6. (Optional:) Set the “CYGWIN=nodosfilewarning” environment variable option to avoid Cygwin warnings about MS-Dos style paths. (Unfortunately, this doesn’t work for me, even though “echo ${CYGWIN}” prints “nodosfilewarning”.)

Note: If you already have Cygwin installed, you can install additional packages by running setup.exe again.

Sound from ByteArray

While writing a quick test application to record microphone input and playing that same recording, I found out that playing back a ByteArray directly using the Sound class wasn’t possible. Luckily a solution is never far away when doing a Google search and it turns out that FlexibleFactory has a good solution: the MP3FileReferenceLoader lib. With a bit of tweaking you can pass in a ByteArray directly and don’t need to rely on a FileReference.

Good news though! As per Flash Player 11, the following 2 methods have been added to the (native) Sound class that can be used to achieve the same behaviour:

  • Sound.loadCompressedDataFromByteArray(bytes:ByteArray, bytesLength:uint)
  • Sound.loadPCMFromByteArray(bytes:ByteArray, samples:uint, format:String = “float”, stereo:Boolean = true, sampleRate:Number = 44100.0)

“java -fullversion” triggers ProgressEvent.STANDARD_ERROR_DATA in AIR

I’m building an AIR app and am using Java to offload some heavy duty work to keep Flash responsive. I want to make sure the end user has a version of Java installed the is compatible with the Jar I ship with my AIR app. To do so I use the following (pseudo) code below:

  1. var args:Vector.&lt;String&gt; = new Vector.&lt;String&gt;();
  2. args.push("-fullversion");
  3.  
  4. var info:NativeProcessStartupInfo = new NativeProcessStartupInfo();
  5. info.executable = javaLocation;
  6. info.arguments = args;
  7.  
  8. javaProcess = new NativeProcess();
  9. javaProcess.addEventListener(ProgressEvent.STANDARD_OUTPUT_DATA, versionOutputHandler);
  10. // Fun fact: -fullversion output is considered error data! :s
  11. javaProcess.addEventListener(ProgressEvent.STANDARD_ERROR_DATA, versionOutputHandler);
  12. javaProcess.addEventListener(NativeProcessExitEvent.EXIT, versionExitHandler);
  13. javaProcess.addEventListener(IOErrorEvent.STANDARD_OUTPUT_IO_ERROR, versionErrorHandler);
  14. javaProcess.addEventListener(IOErrorEvent.STANDARD_ERROR_IO_ERROR, versionErrorHandler);
  15. javaProcess.start(info);

To my surprise the result of the call to “java -fullversion” is send to the standard error stream and thus triggering the ProgressEvent.STANDARD_ERROR_DATA to be fired.

Don’t forget to instantiate Strings

Small wtf moment today. I forgot to instantiate / give a String variable a default value. After appending the string and tracing it, “null” was the first part that showed, see below:

  1. var test:String;
  2. test += "Test 1, 2, 3, 4...";
  3. trace(test);
  4. // output: "nullTest 1, 2, 3, 4..."

Fix:

  1. var test:String = "";

Oops! 🙂

Flex FTP Abort

To properly close a transfer to the FTP server, you need to send an Abort command (see http://www.w3.org/Protocols/rfc959/4_FileTransfer.html (ABORT) ). Flex FTP doesn’t support this so I added some functionality to be able to send an abort.

First I added the Abort command to the Commands class:

  1. public static const ABOR:String = "ABOR";

After that I added a new invoker to the invokers package, AbortInv.

The FTPInvoker class required a new public method to support the Abort command, which can be overridden by the specific implementations. In FTPInvoker I added an abort method like this:

  1. /**
  2. * Stops this Invoker from executing.
  3. */
  4. public function abort():void {
  5. // Some Invokers just don't need to do anything.
  6. finalize();
  7. }

There was only one invoker where I had to override the abort method to get things working correctly and that was the UploadInv class. Here I added the following code:

  1. /**
  2. * @inheritDoc
  3. */
  4. override public function abort():void {
  5. super.abort();
  6.  
  7. if(interval) clearInterval(interval);
  8. if(sourceFile) sourceFile.close();
  9. if(passiveSocket &amp;&amp; passiveSocket.connected) passiveSocket.close();
  10. }

After that it was a matter of updating the FTPClient class with an abort method to get the circle complete.

  1. /**
  2. * Aborts the previous FTP command.
  3. */
  4. public function abort():void {
  5. if(currentProcess) {
  6. if(currentProcess is UploadInv) {
  7. swallowNextResponse = true;
  8. }
  9. currentProcess.abort();
  10. currentProcess = null;
  11. }
  12.  
  13. invoke( new AbortInv(this) );
  14. }

Unfortunately I also had to update handleData method in the FTPClient class a little bit, because aborting an upload would result into 2 (or more) resonses at once from the FTP server.
My handleData method now looks like this:

  1. private function handleData (pEvt:ProgressEvent):void
  2. {
  3. processing = false;
  4. var response:String = ctrlSocket.readUTFBytes(ctrlSocket.bytesAvailable);
  5. var responseList:Array = response.split("\n");
  6. // Apparently we always get a linebreak with a response...
  7. responseList.pop();
  8.  
  9. var eventList:Array = new Array();
  10. var evt:FTPEvent;
  11. for(var i:int = 0; i &lt; responseList.length; i++) {
  12. evt = new FTPEvent(FTPEvent.RESPONSE);
  13. evt.response = FTPResponse.parseResponse(responseList[i]);
  14. eventList.push(evt);
  15. if (FTPClient.DEBUG) trace("-&gt;" + evt.response.code+" "+evt.response.message);
  16. }
  17.  
  18. if(swallowNextResponse) {
  19. if (FTPClient.DEBUG) trace(" - ignoring this response...");
  20. swallowNextResponse = false;
  21. responseList.shift();
  22. }
  23.  
  24. if(responseList.length &gt; 0) {
  25. for(var k:int = 0; k &lt; responseList.length; k++) {
  26. dispatchEvent(eventList[k]);
  27. }
  28. }
  29. }

Links

Flex + Java = Frankenstein?

Lately I’ve been working on a small upload tool in Flex/AIR.
To verify that an upload has succeeded, I created a hash (md5) of the entire file and sent that to the backend for verification. The AS3CoreLib contains a nice MD5 class, MD5Stream, but it’s much too slow for creating a checksum of files.

Actually, it’s so slow that when I was uploading a 5+ MB file the application would slow down so much that the upload speed dropped to a third of the speed without feeding the MD5Stream data during uploading.
Luckily, with AIR 2.0 NativeProcess has been introduced, it made it much easier to “outsource” complicated calculation to a process that can deal with that much better, i.e. Java.

The nice thing about Java is that it’s cross-platform just as AIR, and these days many people will have the JRE installed.  So first of all, I wrote a small Java class that would print out an MD5 hash of a file based on the path that I passed. I jarred the class and added it to my AIR project.
The second step was to call my Java class instead of the AS3 MD5Stream class. This is surprisingly easy. The NativeProcess documentation comes with a pretty clear example and Piotr Walczyszyn has created a small framework, Flerry, to let Flex talk to Java and vice versa.

Looking at how Flerry is built will give you some more insight on how to deal with NativeProcess.
For my application, I only use the BaseStartupInfoProvider class from Flerry to find where Java is installed on the user’s machine. After that I set up all event listeners as per the NativeProcess documentation and call NativeProcess.start() with a NativeProcessStartupInfo as passed in parameter.
Now my uploads are fast again and I can generate a checksum in a fraction of the time I could before.

There are unfortunately some downsides to this approach. You have to deploy your AIR application as native installer to be able to use the NativeProcess class. So instead of having one AIR file, you’ll end up with four native installers (.exe, .dmg, .deb, .rpm). On the plus side, you can create the native installers from an AIR file without needing to resign the application.
Another downside is that now I’m relying on two runtimes being installed on the end user’s machine, AIR and Java.
My final thought is that now that I am outsourcing some calculations to Java, why not built the whole thing in Java? Then again, building a decent UI can take ages using Java. To me, my application feels like I’m creating a bit of a Frankenstein.

All in all I’m glad I now have a fast way to generate a checksum and my uploads aren’t slowed down by it any more.

Links:

Flex FTP EPSV support

Still working with Flex FTP and this time I was missing EPSV support. EPSV is roughly the same as PASV, exect that you only get a port number to connect to back and not an IP address. You can read up on RFC 2428 “FTP Extensions for IPv6 and NATs” if you want more details.

The first thing I did to add support for EPSV in Flex FTP is adding a response to the Responses class.

  1. public static const ENTERING_EPSV:int = 229; //Entering Extended Passive Mode.

Second was the commands class, where I added the following line:

  1. public static const EPSV:String = "EPSV";

Then I added an extra argument to the FTPClient class’ constructor that will indicate whether or not to use ESPV.

  1. public function FTPClient (host:String="", port:int=21, useEpsv:Boolean = false)

I store the value internally and use a getter method to determine if an Invoker class (i.e ListInv) needs to use EPSV.

Based on the client epsv setting any invoker can be updated to send the EPSV command instead of the PASV command when needed, i.e. my implementation of ListInv’s startSequence method:

  1. override protected function startSequence ():void {
  2. if(client.useEpsv) {
  3. sendCommand(new FTPCommand(Commands.EPSV));
  4. } else {
  5. sendCommand(new FTPCommand(Commands.PASV));
  6. }
  7. }

I also update the responseHandler method in ListInv to include the following case:

  1. case Responses.ENTERING_EPSV:
  2. passiveSocket =    PassiveSocketInfo.createPassiveSocket(evt.response.message,
  3. handlePassiveConnect,
  4. handleListing, ioErrorHandler,
  5. null, true, client.hostname);
  6. break;

On top of that I made some tweaks to the PassiveSocketInfo class.
The parseFromResponse method now looks like this:

  1. public static function parseFromResponse (pasvResponse:String, usingEpsv:Boolean = false, hostName:String = ""):PassiveSocketInfo {
  2. var host:String;
  3. var port:int;
  4.  
  5. if (usingEpsv) {
  6. host = hostName;
  7. port = pasvResponse.match(/\d+/)[0];
  8. } else {
  9. var match:Array = pasvResponse.match(/(\d{1,3},){5}\d{1,3}/);
  10. if (match == null)
  11. throw new Error("Error parsing passive port! (Response: "+pasvResponse+")");
  12. var data:Array = match[0].split(",");
  13. host = data.slice(0,4).join(".");
  14. port = parseInt(data[4])*256+parseInt(data[5]);
  15. }
  16. return new PassiveSocketInfo(host, port);
  17. }

The createPassiveSocket method now takes two extra optional arguments (usingEpsv:Boolean and host:String) and just passes those on to the parseFromResponse method.

Hope this helps anybody looking for EPSV support for Flex FTP.

PS: It’s been a while since I added EPSV support, so let me know about any gaps.