Wednesday, January 16, 2019

Re-Implementing Live2D runtime in LÖVE


 (video above shows my implementation in action using LÖVE framework)

Live2D is a nice thing, the fluid character movement gives additional touch to the game which uses it. For my personal use, however it has annoying limitation: Lua is not officially supported, especially LÖVE. Well, there's 2 ways to overcome this.

Writing external Lua C module

This is probably the simplest way (but not that easy). Link with Live2D C++ runtime library files, add code which interact with Lua (and LÖVE), then you got Live2LOVE. This module is actually very fast, considering the Lua to C API overhead, and works by letting Live2D to do the model transformation and LÖVE to do the model rendering. Since it uses official Live2D runtime, it has these limitations:
  • Must link with OpenGL. This is not a problem since LÖVE uses OpenGL already.
  • VS2017 is not supported (you have to use Cubism 3 for that). However it supports down to VS2010, but LÖVE requires VS2013, so this is not really a problem unless you compile LÖVE to use VS2017 runtime.
  • MinGW/Cygwin compilation is not supported. Not really a problem since compiling LÖVE in Windows using MinGW/Cygwin itself is not supported.
  • Linux and macOS is not supported. This is the real problem. Not all people use Windows to run LÖVE.
So another idea that comes to my mind is:

Re-Implement Live2D Cubism 2 Runtime

In Lua, because why not. This is actually somewhat time-consuming process and took me more than 3 weeks to have model rendering working as intended. My additional goal for this is to have Live2LOVE-compatible interface too, so switching between implementation is simply changing the "require" module. From now on, I'll refer Live2D Runtime as Live2D Cubism 2 Runtime.

I start by downloading Live2D Runtime for WebGL (Javascript) and beautify the code (since it's in .min.js file). As expected, the function method names are obfuscated. So, I unpacked Android version of Live2D C++ Runtime and deobfuscate the method name by matching the arguments from Live2D Runtime C++ header files and with help of IDA pseudocode decompiler to compare the implementation with the Javascript ones. This whole process took 2 weeks.

Then I start by writing the Lua equivalent code based on Javascript Live2D Runtime. This is the easiest to do since Javascript is also dynamically typed. Carefully translating 0-based indexing code to 1-based. Then fixing bugs, writing Live2LOVE-compatible interface so I can use existing Live2D viewer code that using Live2LOVE, and testing.

After a week, I got model rendering, motion, and expression working. Using existing code from my LÖVE-based Live2D model viewer to use my own implementation instead of Live2LOVE. Then the next problem comes: it's 4x slower than Live2LOVE. Live2LOVE took 1ms to render Kasumi (casual summer clothes) while my implementation took 4ms to render same model. I already code the implementation carefully so that LuaJIT happily accepts my code and won't bail out to interpreter as possible.

I started optimization by using "Data" objects instead of plain table when updating the "Mesh" object for drawing. This cuts down the update time significantly from 4ms to 1.7ms so using table to update the "Mesh" object is always a bad idea. Someone in LÖVE Discord then says "try to use all FFI instead of plain table". At first I did not agree with him because I want to preserve compatibility with mobile, but then I decide to proceed by having falling back to tables in case FFI is not suitable (JIT is off, FFI support is not enabled, or using vanilla Lua). I swapped most types from plain table to FFI objects and I can get as low as 1.2ms, almost close to Live2LOVE 1ms.

Conclusion

Re-implementing Live2D Runtime is a nice experience. It gives me better overview when to start optimizing code instead of optimizing early and overview of how Live2D model transformation works. Apparently it can't beat C++ version of the official Live2D Runtime in terms of model updating, but I think it can beat it in terms of model rendering. I'm thinking of "Mesh" batching technique, which is basically: accumulate vertices to render then draw'em'all at once if flush requested. I'm currently satisfied with the current result, but I think I still can do better
 

... and hope to God it can success without problems.

Saturday, December 1, 2018

San Andreas "Modern" Modding Tools & Some Rant

When I search for some tutorials in Google about modding GTA San Andreas, it's mostly still uses very old tool. Usually it's Indonesian modding community that always uses old modding tools for "compatibility" reason, but in fact old tools were unstable, and of course newer one means more stable (although may need modern system). For todays blog post, I just want to tell you that there are better tools out there that you can use as replacement. Note that this blog post can be considered as rant, depending on your perspective.

First, IMG editor. This is essential tool for modifying the game models and textures. Usually, the blog post tells you to use "IMG Tool v2.0". Excuse me, there's a way better tool for this, and it's Alci's IMG Editor. Compared to IMG Tool v2.0, Alci's IMG Editor is way more superior. Having bigger windows, import/export multiple files, and most importantly, it can create new IMG file from scratch, unlike IMG Tool, where you only restricted to existing IMG file.

Second, TXD tool. This is must-have tool if you plan to modify texture files (creating your own paintjob). Usually, the blog post tells you to use "TXD Workshop", but excuse me, TXD Workshop is somewhat very limited, and last time I tired the 15 years edition, the mipmap generation is buggy. If you interested how it buggy, my image is 4096x4096, compressed to DXT1, then generate 12 levels of mipmaps. It took around 3 minutes in my i5 7th gen laptop, and the result, it fails to generate mipmap for odd-levels (3, 5, 7, ...). This result in "black shade" when the game render the texture. So let me introduce you to Magic.TXD,  modern version of TXD Workshop, easy to use, and it generates mipmap faster and not buggy of course. It also comes with localization popular among the SA modders including Indonesian.

The texture created with TXD Workshop + mipmap. Level 3, 5, 7, and 11 is buggy and game will render that to black (notice shade of black near the headlights). That's why it looks dark for some reason.
The texture create with Magic.TXD. Now everything looks nice and no more black shade.

For this one, it's not really modding tool, but I want to put it anyway. If you're script modder, then you may want to consider MoonLoader, which is "modern version" of CLEO. Do you know that CLEO scrips are actually slow compared to Lua script? or, you hate CLEO scripts and prefer writing in different language? Then you got MoonLoader. One most notable feature is the error handling. CLEO script errors? Goodbye to current game session. MoonLoader script errors? Error is logged then script execution stopped.

As a side note, if you only want to dump TXD texture to PNG or TGA, FFmpeg supports TXD file and can transcode it to supported image formats (but you can't create TXD sadly).

Sunday, October 28, 2018

Music quality is associated with file size. True?

People often say that "if the audio size is big, that means the quality is nice". Actually, not really. Let me tell you one true story.

One day when I was surfing with my Facebook, there's Romeo and Cinderella movie video which was performed by Poppin'Party. It was part of BanG Dream x VOCALOID collaboration, so yeah it's normal, except the movie video uses the full version of the song. With my element inspection & dev console magic, I managed to get the audio file (it's not really legal, forgive me). The audio is encoded using AAC-HE encoding, with 48Kbps bitrate and frequency at 44100/22050Hz. The resulting file size is ~1.6MB. If you think the quality is bad, keep reading the story below.

A few days later, one of my friend managed to get 320Kbps MP3 version of the song, and posted a screenshot about it. The screenshot display full artist name, title, and even cover art. However, as you may know, those can be easily edited with many free software (FFmpeg allows you to add artist name and title, and the cover art can be easily obtained from clickable "Poppin'Party" above). When I see his post, I was skeptical about it. so I managed to ask if he can send me the audio file for analysis purpose, in exchange with the AAC-HE version that I have. He lend me the MP3 version, which has size around ~10MB (which is normal for 320Kbps MP3), and I also lend him my AAC-HE version, which is ~1.6MB (almost 10x smaller). He can't tell the difference between his MP3 version and my AAC-HE version sadly, but he only says that mine has lower audio volume (which is not really related).

Now, it's time to put it to FL Studio's "Wave Candy" plugin. First, I set up FL Studio to listen to Stereo Mix (it's rare to find laptop/PC with this feature). First test is to use my AAC-HE version that I got from Facebook. The result? bit surprising. I already knew that AAC-HE has better quality for lower bitrate as low as 32Kbps. The AAC-HE version has ~16KHz fieldity (better band preservation). Then when I test the MP3 file, my guess is correct, the band preservation is only ~12KHZ, way lower than AAC-HE version that I got. I also can differentiate the audio quality with my ear alone, and my ear tells me that the MP3 audio quality is lower than the AAC-HE version.

From that, we can conclude that smaller audio file size doesn't mean lower quality. It depends on encoding used and how lossy it was before re-encoded. Because, as you may know, re-encoding lossy audio degrades the quality even and even more. You also can't trust the file size alone because, say I re-encode that Romeo and Cinderella song (AAC-HE version) to WAV 44100Hz (which yields to ~40MB), I won't gain anything, and in fact I only waste drive space (it will only have ~16KHz fieldity). The quality is then exactly same as AAC-HE version, and if I re-encode the WAV to AAC-HE again, the quality won't be same as the first one.

Then, back to the title of this post. Music quality is associated with file size. True? The answer is FALSE.

Friday, October 19, 2018

Girls Band Party! PICO Episode 16 Bodyswap Explained.

Before you read the rest of the post, please watch the video.

Okay based on that video, we can see that all members of Poppin'Party is switching bodies to each other. This blog post will explain who swap to which one.

  1. Duration 0:41, we can see Kasumi's sister, and Arisa saying "I'm home, Acchan" (Kasumi call her sister Acchan). Then Kasumi's sister asking "Ichigaya-san where's my sister" then Arisa says "What are you talking about?". Also at duration 1:03, Arisa replies "This is Kasumi". So, we can then conclude that Kasumi is in Arisa's body.
  2. The following duration (1:03), we can see it's not usual Rimi expression (she's usually shy), which is more similar to Arisa. Also in duration 1:59, Rimi is asking Arisa (which is Kasumi inside Arisa body) to bring her body back. From this, we can conclude that Arisa is in Rimi's body.
  3. Duration 2:02, Saaya asking Rimi to stop bang her head to Arisa followed by "Stop! That's my body!". For some people, it may sounds ambiguous, considering Kasumi is in Arisa's body now, and Arisa is in Rimi's body, also with different face expression of Saaya, I think we agree that Rimi is in Saaya's body.
  4. The rest (Tae and Saaya) is bit hard. We'll start with Tae first. Where's she? Let's take a look at the beginning, duration 0:15. Remember Tae's expression correctly here. Then seek to duration 2:04 (if possible, slow the video down to 25% to see correctly). You can see Kasumi's expression is similar to Tae one in duration 0:15. Based on this, I think we can agree that Tae is in Kasumi's body.
  5. Then, where's Saaya? I think it's clear which body is Saaya. Kasumi is in Arisa's body, Arisa is in Rimi's body, Rimi is in Saaya's body, Tae is in Kasumi's body, then Saaya must be in Tae's body!
I think that's it. If you still don't understand, please see this table.

Body Kasumi Tae Rimi Saaya Arisa
Soul Tae Saaya Arisa Rimi Kasumi

When you promote your band, make sure to promote your band at pet shop at first.

Friday, June 29, 2018

LÖVE Optimization Tips

It’s common for people to write game and then “damn it performs very slow”, or “ok this game is fast enough” then when testing in slower hardware it complains “damn it’s slow af”. So these points will help you to optimize your game for more performance. This assume you’re using LOVE with LuaJIT as LuaVM. If you have different configuration (like LOVE with Lua 5.2 or with Lua 5.1), some points in here may not relevant.

Note that these things may be considered as micro-optimization in desktop PC, but it’s noticeable in mobile.

Move your game computation in love.update, and keep love.draw solely for rendering.


Okay let’s see why. First, math computation in LuaJIT are compiled, which mean it’s fast. Second, calls to love.graphics api in LuaJIT are not compiled and break the code around that area to be not compiled. So instead of

function love.draw()  
 var1 = var1 + 20 -- this computation is done in LuaJIT interpreter  
 -- because calls to love.graphics.draw below can't be compiled  
 love.graphics.draw(image, var1 * 0.02, 0)  
end

Do it like this.

function love.update()
 -- this computation is done in machine code because
 -- it's compiled.
 var1 = var1 + 20 * 0.02 -- LuaJIT will optimize 20*0.02 to 0.4  
end

function love.draw()
 love.graphics.draw(image, var1, 0)  
end

A note: This is less relevant for mobile (iOS and Android), as JIT compiler is disabled by default.

Use SpriteBatch, especially if it’s static.


Admit it. All calls to SpriteBatch object method will be never compiled. As of LOVE 11.0, there’s a feature called autobatching. That’s it. It automatically batches your draws, much like SpriteBatch with “dynamic” and “stream” mode.

Now, why use SpriteBatch for static batches? That’s because it’s better to call love.graphics.draw once to draw many things than calling love.graphics.draw to draw each thing. The former saves CPU due to Lua to C API call overhead, allowing your game to run faster.

Only use pairs when you don’t have a choice.


Even with LuaJIT 2.1 trace stitching feature (allows uncompiled function like love.graphics.draw not to break JIT compilation), pairs still breaks JIT compilation because it uses uncompiled bytecode. You can check what things that are compiled or not in here.

If you really need to index by things other than number, store the key in numered array, index that, then use the value to index the actual table. Then you can use ipairs or simple for loop to iterate the table (both are compiled). Something like this.

myTableKey = {"key1", "key2","key3"}
myTableValue = {key1 = "Hello", key2 = "World", key3 = io.write}
for i, v in ipairs(myTableKey) do
 local value = myTableValue[v]
 -- do something with `value`.
 -- It can be either "Hello", "World", or `io.write` function.
end

Oh also pairs doesn’t guarantee the order they’re defined. When you run pairs in myTableValue above, key3 might be shown first, then key1, then key2.

Use io.write for simple debugging, not print.


print is not compiled, but io.write is. The downside of io.write is that you’ll lose of automatically separated values and object-to-string conversion feature. If you don’t care about this, you can use print. Well, this is solely a preference. If you need the full power of print, then use print. For very simple variable printing, use io.write.

Never use FFI for computation optimization purpose.


Very true if you’re targetting mobile devices. Check out this chat I took from LOVE Discord server.
A: Will vector-light be faster than brinevector?
B: Depends. Brinevector is faster, but only on desktop. It’s terribly slow on mobile. Vector ligh (hump?) Is faster, but the code is messy since thats only for vector operations
A: if I made an FFI version of vector-light that was just pure functions, it would be faster right?
That’s where the problem begins. Once you step into FFI, your phone will start to cry when running your code.
Wait, based on benchmark, FFI is faster than plain Lua table.
That kind of argument doesn’t work once you have JIT compiler disabled. I’ve tried to benchmark it that in most cases FFI is ~20x slower than plain Lua when JIT compiler is disabled. Then what’s the matter for mobile devices? Take a look at first tips above. Mobile devices have JIT compiler disabled by default, which means slow FFI.

Well, you shouldn’t use FFI in mobile devices at all anyway. Starting from Android 7, any calls to ffi.load will just fail due to changes in Android 7 dlopen and dlsym which restrict those function greatly that it makes ffi.load cease to function. And in iOS, I don’t see the point of using FFI as all things must be statically compiled.

Switch the JIT compiler before love.conf is called in conf.lua


You may ask yourself. Why? The answer is, LOVE utilize some FFI call optimizations when it detect it runs with JIT compiler enabled, mainly in love.sound and love.image modules, and the detection is done before main.lua is loaded. Some people did this wrong by turning the JIT compiler on/off in main.lua, which can lead to slower performance.

Conclusion


Those kind of optimization is mostly helps in CPU optimization. For GPU, you need to reduce drawcalls, reduce canvas usage (or don’t use it at all), and use compressed texture. If it’s still slower, but the CPU stays low, that indicates you’re fillrate-limited, in which case, any optimization in here doesn’t really help you at all.

Note that I may still wrong, so if you have something to add, or something is invalid in above tips, just comment below to discuss it.

'What was the original problem you were trying to fix?' 'Well, I noticed one of the tools I was using had an inefficiency that was wasting my time.'

Also, did I also mention that JIT compiler for ARM64 is unstable?