FL Studio and FFmpeg Libraries

Did you know that under the hood, FL Studio uses FFmpeg for some of their operations? For example, the Fruity Video Player plugin uses FFmpeg to load wide array of video codecs. And ZGameEditor Visualizer export function uses FFmpeg libraries for its video encoding.

Is this good or bad? Both. It's good because FFmpeg is the mother of codecs and formats, so it can decode lots of audio and video formats. But it's bad because FL Studio's bundled FFmpeg libraries are LGPL which lack some video encoder like x264. This causes problem where ZGameEditor Visualizer plugin lossy encoding option exports to H263 when using .mkv.

Note: ZGameEditor Visualizer seems always export to PNG in MP4 if you choose MP4 regardless of the "Uncompressed video" option.

So is it possible to change the old H263 to H264? Yes. LGPL software must be dynamically linked so it's user-replaceable, and fortunate for us, FL Studio dynamically links to FFmpeg. So how to replace the libraries? Just follow the steps below. Note that I assume 64-bit version of Windows and you're using 64-bit version of FL Studio executable.

1. Make sure FL Studio is not running.
2. Download 64-bit Windows FFmpeg shared binaries (release essential or release full). If you already have it downloaded before, you can use that. Just make sure it's latest version.
3. Navigate to "%ProgramFiles(x86)%\Image-Line\Shared\ffmpeg\x64" and replace all the DLLs (except ILVideo_x64.dll) with the DLLs from the downloaded zip. Just the DLLs, not the exes too. It's good idea to backup all the DLL files there, just in case.

UPDATE: If you come here before, you'll notice I changed the link. This is because Zeranoe no longer provides FFmpeg binaries.

 Now start the FL Studio again, export to .mkv and it will use H264 (x264).

So what about other video codecs? Unfortunately FL Studio specialize MP4 extension and forces PNG in MP4 when used. But from my experiment, here are list of possible extension to use and the resulting video codec:

• .mkv - H264 video codec (high profile) and Vorbis audio codec
• .webm - VP9 video codec and Opus audio codec
• .ogv - Theora video codec and Vorbis audio codec
• .nut - MPEG4 video codec and AAC audio codec

Any other benefits of replacing the DLLs? Not sure if this another feature is caused by replacing DLL, but you can basically load any audio as long as the extension is one of FL Studio can recognize. Sadly that means you have to change the extension to .wav, .flac, .wv, .ogg, or .mp3 first before loading it to FL Studio.

I just hope the ZGameEditor Visualizer export function has more option of controlling the video output when using custom presets. I'd love that.

Container vs. Codec

Often people misinterpret container as video format. Here's one simple scenario.

Say, there's A who have very old phone, released in 2007, and B who have smartphone released in 2019.
A: Can you send me that MP4 video in your smartphone
B: Sure.
A: Why it doesn't play in my phone? The phone says it supports MP4 video format but it can't play your video. Your video is bad and so do you.

So let's make this clear. First of all there are no such thing as "MP4 video format". Keep your questions for later. In the world of multimedia files such as video and audio, you have to know 3 things:

1. Container file
2. Audio codec
3. Video codec

Container is a file that stores information how the video and the audio are stored inside the file. Sometimes also subtitles and additional files. Container also has information about what decoder should be used to decode the video and the audio and also contains when those video and audio should be decoded and presented to user. Here are some popular container file:

1. MP4. Yes MP4 is a container, not a video format. Now you know!
2. WebM.
3. Matroska. Also known as MKV. The mother of container as it supports almost every codec in existence.

So, if MP4 is not video format, what are actually video formats? That's what called video codecs. It represents the data about how the video are encoded, basically video codec is what important about compatibility. Even if, say I use MP4 which was supported in A's phone above, but I use more recent codec, then A's phone won't able to play the video, despite being MP4.

So what are kinds of video codecs?

1. MPEG4, XviD/DivX family goes here. A very old codec.
2. H264, most popular codecs since smartphone existed.
3. H265, recent codec which provides smaller size and better quality.
4. VP9, royalty-free codec by Google which compete with H264.
5. AV1, royalty-free codec various vendors which compete with H265.

PS: "royalty-free" term doesn't mean anything from user perspective. It only matters from developer perspective.

For scenario above, if B's video contains H265 codec, then A's phone won't able to play it, even if the video itself is inside MP4 container file. Now everything makes sense, right?

Then there's also audio codec. Watching silent video is not very fun right? Then here comes the audio codec. The definition is same as video codec above, but for audio instead. There's only one difference, most audio codecs can be extracted out of their container, being standalone file. That's not possible for video codec.

So, list of audio codec please? Okay.

1. AAC. Its standalone file extension is .aac (actually MPEG ADTS). Can be placed inside MP4 container.
2. Opus. Must be placed in Ogg or WebM container.
3. Vorbis. Must be placed in Ogg container.
4. FLAC. Its standalone file extension is .flac. Can be placed inside Ogg container.
5. MP3. Its standalone file extension is .mp3. Can be placed inside MP4 container.

Whoa, hang on, so Ogg is not an audio format? Yes. Ogg is also a container. It can contain Theora video codec.

So the conclusion is, if the video file is in extension that you know, that doesn't guarantee your device can play it. Like, you feel your device is superior because it can decode AV1 in MP4 until FL Studio's ZGameEditor Visualizer lossless video export function writes PNG image inside MP4.

Know Your Lua Errors

One problem when trying to help people with their Lua problem is that they don't know how to read Lua errors. You can be 100% sure that saying "it doesn't work" is absolutely not helpful at all. So I'll tell you the ways to read Lua errors properly and what it means.

In this post, I used "<>" symbol to denote something arbitrary. Start by reading the syntax of Lua error. Lua error is usually have syntax like this:

<filename>:<lineNumber>: <message>

<filename> is the script file. <lineNumber> is line number where it occured. That's very easy to remember, right? Now for the error message. The error message is usually very descriptive but sometimes it doesn't really tell you what exactly is wrong, so here are some lists of Lua errors that I'm aware of. If your Lua error is unlisted, that can mean I didn't add it yet or it's thrown by external program.

1. attempt to call global '<name>' (a <non-function> value)
   This caused when you're trying to call a global variable called '<name>' but '<name>' is not a function type. Example
   

   print() -- works
   table() -- attempt to call global 'table' (a table value)
   _VERSION() -- attempt to call global '_VERSION' (a string value)
   anilvalue() -- attempt to call global 'anilvalue' (a nil value)
   

2. attempt to call field '<field>' (a <non-function> value)
   Similar to above, but this occurs when you try to call something within a table.
   

   -- Note that 'math' is a global variable which is table
   math.abs(123) -- works
   math.pi() -- attempt to call field 'pi' (a number value)
   

3. bad argument #<n> to '<name>' (<type1> expected, got <type2>)
   This is caused when a function '<name>' expects value with type <type1> for n-th argument, but user passed something with type <type2> instead.
   

   -- io.open expects string for the 1st argument
   local file = io.open(io.open) -- bad argument #1 to 'open' (string expected, got function)
   -- tonumber 2nd argument expects number if present
   tonumber("0xFF") -- works
   tonumber("0xFF", table) -- bad argument #2 to 'tonumber' (number expected, got table)
   

4. table index is nil
   To be honest, this is most undescriptive Lua error message. What it means that you try to assign a value to a table at index "nil" (I mean literal nil).
   

   table[nil] = io -- table index is nil
   

5. bad argument #<n> to '<name>' (invalid option '<something>')
   This means you passed invalid option. Notable function that throw this is 'collectgarbage' and 'file:seek'.
   

   print(collectgarbage("count")) -- works
   collectgarbage("asd") -- bad argument #1 to 'collectgarbage' (invalid option 'asd')
   

So I think that covers most common Lua errors that you mostly encounter. In case you need help, please provide the Lua error message, plus the traceback if available. The traceback is also easy to read, the syntax is similar to above, and it definely helps.

... unless you got a very rare "PANIC" error which is unhelpful. No, really.

Enabling Private DNS for Modified-Android that Lack Such Settings?

One of the best features in Android 9 is Private DNS feature which allows DNS request not to be modified in any way by third-party or even by your ISP (this is always in Indonesia huh). Basically it does encrypt your DNS request and sign it so no one (except destination) can see it. And even if they see it, they can't modify it because it's signed.

Enough for that, some phones running Pie unfortunately lack such features. Some OSes like MIUI actually just hid them, so invoking "am start com.android.settings/.Settings\$NetworkDashboardSetting" will show them. However, something like ColorOS completely removed it from their settings. So we shouldn't rely on that MIUI method.

Now my idea is "What if we set those options from ADB instead?". It took me some research and ADB shell + grep with my phone and here's what I found.

$ settings list global | grep dns
private_dns_mode=hostname
private_dns_specifier=1dot1dot1dot1.cloudflare-dns.com

It looks clear that we can simply set those values from ADB. The shell does have access to modify those settings, at least in my phone (Mi A1 as of writing). So here are the possible combination of setting.

$ settings put global private_dns_specifier resolver_hostname
$ settings put global private_dns_mode off|opportunistic|hostname

Change "resolver_hostname" to something like "1dot1dot1dot1.cloudflare-dns.com" and see if it works. Note that the Private DNS hostname setting only works if "private_dns_specifier" is set to "hostname". If your phone stops connecting to internet (can't resolve any hostname), that means you messed up the "private_dns_specifier". Double check and try again.

Note that this method works in my phone with ability to set those options in the UI too, so it would be good if someone can test this in phones that running Android 9 but lack that option in their settings UI.

Update: If you get something like "Neither user 2000 nor current process has android.permission.WRITE_SECURE_SETTINGS" that means the OS customizations enforce some additional protection. You may (or may not) able to disable those settings in developer options window too and try again. Thanks to my friend for testing this in Realme 3 Pro, the feature actually work as intended.

MediaFoundation decoder for LOVE & decoding in-memory.

Windows 7 has its own COM-based API to aid decoding variety of audio formats, and if you concerned about patents (AAC, why), don’t worry as it’s a licensed decoder. This blog post is about how I wrote LOVE decoder that uses MediaFoundation.

Basics

First thing to know is how LOVE decoder class looks like.

class Decoder : public Object
{
public:
    static love::Type type;
    Decoder(Data *data, int bufferSize);
    virtual ~Decoder();
    static const int DEFAULT_BUFFER_SIZE = 16384;
    static const int DEFAULT_SAMPLE_RATE = 44100;
    static const int DEFAULT_CHANNELS = 2;
    static const int DEFAULT_BIT_DEPTH = 16;
    virtual Decoder *clone() = 0;
    virtual int decode() = 0;
    virtual int getSize() const;
    virtual void *getBuffer() const;
    virtual bool seek(double s) = 0;
    virtual bool rewind() = 0;
    virtual bool isSeekable() = 0;
    virtual bool isFinished();
    virtual int getChannelCount() const = 0;
    virtual int getBitDepth() const = 0;
    virtual int getSampleRate() const;
    virtual double getDuration() = 0;

protected:
    StrongRef<Data> data;
    int bufferSize;
    int sampleRate;
    void *buffer;
    bool eof;
};

Anything that ends with = 0; means pure virtual method which we must implement in our derived class. Now let’s derive the Decoder class.

MFDecoder Class

class MFDecoder: public Decoder
{
public:
    MFDecoder(Data *data, int bufferSize);
    virtual ~MFDecoder();

    static bool accepts(const std::string &ext);
    static void quit();
    Decoder *clone();
    int decode();
    bool seek(double s);
    bool rewind();
    bool isSeekable();
    int getChannelCount() const;
    int getBitDepth() const;
    double getDuration();

private:
    static bool initialize();
    static void *initData;
    // non-exposed datatype to prevent cluttering LOVE
    // includes with Windows.h
    void *mfData;
    // channel count
    int channels;
    // temporary buffer
    std::vector<char> tempBuffer;
    // amount of temporary PCM buffer
    int tempBufferCount;
    // byte depth
    int byteDepth;
    // duration
    double duration;
};

There are few points that must be noted here.

• MFDecoder contstructor receives LOVE Data object, which is data located in block of memory, and the desired buffer size.
• MediaFoundation API can return any number of samples so we use temporary buffer to contain leftovers after decoding.
• You may notice that mfData is void*. The reason to do this is that there’s problem compiling LOVE if Windows.h is included BEFORE the keyboard module. We also don’t want to clutter the includes with Windows-specific includes and drag down the compilation time.
• The tempBuffer is std::vector. Yes this is intentional so we don’t have to manage the allocated memory and take advantage of RAII. It also helps in case MediaFoundation returns data bigger than provided buffer by simply reallocating bigger temporary buffer.
• Then there’s initData member. This is set at initialize function above it, which is called when new MediaFoundation decoder is created or if the compatible extensions are being checked.

Problems

Now there are some problems.

• MediaFoundation doesn’t officially support loading media from memory.
  MediaFoundation assume you have the media file reside somewhere in local filesystem or in network, probably due to their DRM nature. There’s this blog post, but it only works on Windows 8 and using C++/CLI which has its own 2 problems:
  
  1. We can’t use C++/CLI when compiling LOVE, that would reduce our compilation times and increase the bloat which we want to try to minimize as possible.
  2. My target is to make the decoder available for Windows 7 and later. IRandomAccessStream and the function it uses are Windows 8 and later.
  Then I found out there’s MFCreateMFByteStreamOnStream which accepts IStream interface. IStream interface is available since Windows 2000, which then SHCreateMemStream can be used to create one.
  
• The functions that I use requires linking to Mfplat.dll and Mfreadwrite.dll.
  The latter is only available in Windows 7. Since I want to make sure it runs in Windows Vista too (without the MediaFoundation decoding capabilities of course), I have to dynamically load it, hence the MFDecoder::initialize() static function.
  
• Once IMFByteStream is created, you have to set the MIME.
  It’s done by casting the IMFByteStream to IMFAttributes and set the MIME. Unfortunately, as of LOVE commit ccf9e63, the decoder constructor no longer receives the audio extensions, so we have to test for every possible supported MIME types. Fortunately, Microsoft gives us list of supported media formats in their documentation, so I just get the MIME string from IIS MIME Types.
  

After those problems is resolved, it’s only matter of setting the properties of the IMFSourceReader like creating decoder which outputs PCM.

Seeking

Yes, the IMFSourceReader supports seeking, but there’s no guarantee that it will be accurate. The function you’re looking for is IMFSourceReader::SetCurrentPosition which accepts time as 100-nanoseconds (second to 100-nanosecond is multiply by 1e+7).

Another Problem: GUID

I’m getting linker errors when compiling LOVE with the MediaFoundation decoder as it complains about unresolved GUID. I also don’t want to link to any of MediaFoundation DLLs so it’s binary compatible with Windows Vista (XP support is dropped as of LOVE 11.0). Temporary workaround to fix this is to copy the GUID declaration in header files to const GUID variables.

Aftermath

After I get everything running, now I have LOVE build which can loads AAC and WMA using MediaFoundation, how good is that, huh? You can check the full source code in here. The respective header lies in same directory as the C++ file.

Now you may ask, what about MinGW? Well, unfortunately LOVE doesn’t support being compiled under MinGW in the first place, so compiling LOVE under Windows is only supported using MSVC compiler.

And if anyone wants to decode audio from memory using MediaFoundation, then this blog post is what he/she’s looking for.

Post is written in Markdown first then converted to HTML lol.