Build Your Personal English Learning Assistant with Android

I want to improve my English, one way to do this is watching documentaries on TV, but subtitles distract me from the video themselves, I get this around by masking them with a always on top window placing at the bottom area.

I named this project as Pela, the Personal English Learning Assistant.

Always on Top Window

Speaking of the top window, there are plenty of options out there, like facebook like Chathead, or libraries like StandOut.

For simplicity I prefer the former one, the final work looks like this:

First create a project with Android Studio, then add permission and our service to AndroidManifest.xml

<uses-permission android:name="android.permission.SYSTEM_ALERT_WINDOW" />
    <service android:name=".ELAssistantService" />

Create a service named ELAssistantService, overrides method onCreate() to add view to the WindowManager:

public void onCreate() {
    super.onCreate();
    mView = new LinearLayout(this);
    mView.setBackgroundColor(Color.GRAY);
    LayoutParams params = new LayoutParams(
            width, height,
            LayoutParams.TYPE_SYSTEM_OVERLAY,
            0,
            PixelFormat.TRANSLUCENT);
    params.gravity = Gravity.CENTER | Gravity.BOTTOM;
    params.y = 20;

    WindowManager wm = (WindowManager) getSystemService(WINDOW_SERVICE);
    wm.addView(mView, params);
}

The last step is to start service in MainActivity.java:

startService(new Intent(getApplication(), ELAssistantService.class));

Launching this app will lead to a AndroidRuntime exception:

10-19 11:21:03.829  8203  8203 E AndroidRuntime: java.lang.RuntimeException: Unable to create service com.bay.elassistant.ELAssistantService: android.view.WindowManager$BadTokenException: Unable to add window android.view.ViewRootImpl$W@d344ffc -- permission denied for this window type

To get this working, SYSTEM_ALERT_WINDOW permission needs to be granted, best way to do this is to prompt user to accept it, but for some android systems that lacks of SystemUI.apk, this is not happening, the following command do the trick:

adb shell appops set applicationId SYSTEM_ALERT_WINDOW allow

And the applicationId can be found in app/build.gradle:

ELAssistant rg applicationId
app/build.gradle
7:        applicationId "com.bay.elassistant"

Issue this command: adb shell appops set com.bay.elassistant SYSTEM_ALERT_WINDOW allow and launch the app to start watching now.

Start Service on Boot Completed

This is optional, for convenience I want it show up automatically, so I don’t need to launch this app when I want to watch TV, add these to AndroidManifest.xml:

<receiver android:name=".ELAssistantBroadcastReceiver">
    <intent-filter>
        <action android:name="android.intent.action.BOOT_COMPLETED" />
    </intent-filter>
</receiver>

And create a service(ELAssistantBroadcastReceiver.java) to handle boot completed intent:

import android.content.BroadcastReceiver;
import android.content.Intent;
import android.content.Context;

public class ELAssistantBroadcastReceiver extends BroadcastReceiver {

    @Override
    public void onReceive(Context context, Intent intent) {
        context.startService(new Intent(context, ELAssistantService.class));
    }
}

Future work

The next step I want to do is to make the floating window self-adaptive according the position of the subtitles detected, for doing this the first thing to do is contour detection. Lars Nieradzik has already did this in his project chinese subtitle ocr using SSD(Single Shot MultiBox Detector), what I need to do is make this works on Android, this tutorial for opencv could be a good starting point.

References

• Creating a system overlay window (always on top)