Removing SIM PIN using AT commands

Ever needed to use a sim card/modem in a headless device which doesnt support a PIN code? (cheap chinese gsm-enabled sensors, I’m looking at you!) You can remove it with a simple AT command:

After that, the SIM card will work without a PIN code (yes, yes, security, I know, I know, but with prepaid cards, who cares)

Sound pitch fix with fast-forward in mplayer

Have you ever watched a video on Youtube using a faster play speed (1.25x, 1.5x,..) and the sound seemed “the same but faster”. Ever did the same (using [ and ] keys) in mplayer, and all the voices were higher (high-pitched), and hard to listen and understand? There’s a filter in mplayer that fixes that and it’s called scaletempo.

mplayer -af scaletempo /foo/video.mp4

Now you can change the playback speed without the annoying pitch changes!

Enabling SSH on a fresh Raspbian Jessie image

This post is mostly a reminder for myself, because I will probably forget this (again) soon.

Sometimes you want to use a RaspberryPi as a headless machine, so first you download the newest Rasbian image, ‘dd’ it to the SD card, insert the SD card into your RaspberryPi, connect the ethernet and power cables, check the IP in your DHCP server logs, run ‘ssh <ip>‘ and..

..connection refused!

I have no idea why, but the ssh server is disabled by default (probably a security feature, since the default password is easy to guess, but still..). Then you start googling, and first couple of links all say that you need to connect the keyboard, display, log in localy, etc. etc. But you’re lazy, and want to do it directly on the PC when you write the image to the SD card. So the easiest way is, after you’re done with ‘dd’-ing, you mount the first partition (a smaller, FAT partition containing the boot files), and just create an empty ‘ssh’ file (with eg: touch ssh) in that folder.

If you also wish to enable wifi (which is enabled by default, but no configurations are stored), you also need to mount the second (ext4) partition, containing the full system – there you can edit ‘./etc/wpa_supplicant/wpa_supplicant.conf‘ file, and add your wireless configuration

After that, sync, umount, insert the card, connect all the cables, and it works!

Mounting VirtualBox VDI images (and similar formats)

Sometimes you have a .vdi (or .vmdk or similar) virtual machine disk image, and you wish to access files inside, without starting the virtual machine itself and copying files via network or some ‘guest additions’ tools provided by the virtualization software.

To do this, you need nbd support enabled in the kernel (and module inserted) and qemu installed (which is actually another virtualization software, but we need it for the qemu-nbd utility).

In some distributions, the nbd module is enabled by default, on some you have to enable it when compiling the kernel sources.

Due to some weird reasons, the nbd module by default supports zero (0) partitions inside a block device:

So when inserting the module, set the parameter to a larger number (eg. 4 or 16 or whatever)

After that, you use the qemu-nbd utility to ‘connect’  the vdi image to the nbd block device (qemu-nbd actually works as a block device server), mount the partition, do stuff, umount the partition, and disconnect the qemu-nbd:

 

Checking how much data still needs to be written when sync-ing on Linux

Sometimes on Linux, when you copy alot of data to a slow USB flash drive, the operation is “done” in a second, but when you umount the drive (or run ‘sync’) it takes forever to write the cache to the actual device.

So, how to check the status (how much time is left, how much does it still need to write)?

The answer hides in /proc/meminfo. Specifically the “Dirty” and “Writeback” lines in that virtual file. Very simplified, the “Dirty” value tells you how much data is waiting in cache, and the “Writeback” tells you how much data is being currently written to the device (chunk size).

You can check both values in semi-realtime with the following command:

 

Measuring frequency with a BusPirate

I recently ordered a NE555 pulse generator from DX with some evil plans in mind, and I had to set the frequency to the desired value. Since calculating the frequency by hand and guessing the number of turns on the cheap pots was not an option, I had to measure it.

Since my multimeter only goes to 1000Hz, I decided to use my BusPirate (it goes to a couple of MHz). I’ve connected the GND and 5V leads to GND and VCC pins on the module, and the AUX lead to the modules output (BusPirate measures frequency on the AUX pin/lead).

BusPirate + ne555
BusPirate + ne555

After connecting the BusPirate to my computer via USB, and opening the serial port (“/dev/ttyUSB0”, 115200 8N1 in my case), i see the BusPirate interface. By default it stays in “HiZ” mode after being plugged in (high impedance mode on all pins – to protect the connected circuits). Pressing the ‘?’ key and Enter, you get a help printout:

If i press the ‘f’ key and Enter the frequency measurement starts, but it measures 0Hz (since the power supply to the module is off). To turn the power supply on, i need to use the “W” key, but this doesnt work in HiZ mode:

To use the power supply, I had to select some other mode (not HiZ) by using the “m” key, so I picked UART (3), and some default speed settings (which don’t matter now, since I wont actually use UART).

In UART mode, I can now turn the 5V power on, and measure the frequency:

Then I’ve tweaked the potentiometers, ran the measurement (“f” + Enter) again, and repeat until the desired frequency is set.

Browsing the Slovenian vehicle database

Slovenian “National Interoperability Framework” (NIO) published a database with data on every registered vehicle in Slovenia, so I decided to do some fun queries. (Query results are in Slovene, but i will try to translate the most interesting ones)

How many registered vehicles are in Slovenia?

What about different vehicle types?

Common manufacturers (cars only):

Since data entry is done at registration points by hand, you see “MERCEDES BENZ” and “MERCEDES-BENZ”  etc. as different manufacturers.

How many Lamborghinis (all vehicle types, all spellings)?

Nice! Must be alot of rich people here.. but lets check the vehicle type first:

Okay, maybe not. Lets check the not-tractors:

Nice, 4 actual Lamborhini cars.

What about some demographic data – how old are the owners (Starost == age)?

In Slovenia it is legal for a minor to own or inherit a vehicle, but to register it, an adult “user” with the appropriate licence must be assigned.

Which car does to youngest car owner have? (not sure if it’s inherited or a data entry error – age:=1)

How many cars are registered per administrative unit (geographical)?

 

What about fuel type (gasoline vs diesel vs others)?

Most common engine size/displacement (ccm):

 

And colors?

 

I will probably add more queries/results, when I get some new ideas and/or questions.

Converting data from Microsoft Access (mdb) format to MySQL

Slovenian “National Interoperability Framework” (NIO) publishes some interesting data every now and then, but most data is usually in MS Access ‘mdb’ format and/or ‘csv’. Working with csv files is painful, since you have to do everything by hand (no selects, filters etc, you have to go through each and every line and check the conditions in a giant loop), and using Microsoft Access on Linux is an even bigger NO.

So, i decided to download some data (anonymized data on drivers licences in Slovenia), and convert it from mdb to MySQL (and insert it into the database). I could have done the same with parsing the csv file, but doing it from mdb autogenerates the database schema, (with csv, you have to do that yourself, by hand).

For converting the data i used mdbtools. The data i used can be found here (zipped mdb).

The first thing after downloading and unzipping the data, was to check which (and how many) tables are inside – you can do this with the “mdb-tables” command.

So, there is one table in the database named “Vozniki_2013” (old data, I know).

To generate the schema I used the “mdb-schema” command (with MySQL output):

Every field is “varchar (510)”, and that’s long. I’ve checked the data, and it fits into “varchar (50)”, so you can do a sed replace, to change the field size in the output. After that, you connect to the database, paste this output into the console (or pipe it directly from the command), and you will have an empty table created for your data.

This would also be a good time to create any indexes you need.

Now all we need is the actual data; we can get that using the “mdb-export” command (we have to tell it the table we want to export, and the output syntax).

All you have to do now, is to redirect this to a file, and copy it to te server with the database, and import it (or pipe it directly, if you can use the MySQL client on the machine you’re using now).

If you have a large table with alot of lines/INSERTs, you will notice that the import takes alot of time. The main slowdown for me was because MySQL commits the data after every INSERT statement, and that takes time. If you want to speed it up, you can disable the autocommit, and manually commit at the end. Since i’ve exported the data to a text file first, I just added the two lines in the beginning and the end of the file:

And voila, you have all the data in your MySQL database now.

Fixing video tearing on Intel Broadwell graphics

I have a Thinkpad X250 laptop (Core i7 5600U CPU) with integrated graphics. When I watch video, in some cases (usually window mode + flash player) causes very noticable video tearing effect (almost like interlacing). It usually didn’t bother me (I mostly watch video fullscreen, where there is no tearing).

And the fix? More obvious then i though. Create a file /etc/X11/xorg.conf.d/15-intel.conf with this inside:

Yeah, who would have thought I had to enable the "TearFree" option…

Merging multiple ESRI SHP files into one

I’ve been playing around with GIS software for a couple of weeks now, and i’ve seen alot of datasources using tiled .shp files (instead of all the data in one huge .shp file, the data is split into “square” tiles). If you want to do stuff with that data, you sometimes want it in one .shp file, so here is a very simple script to merge the files to one:

It just does a for loop over all shp files, and uses GDAL to merge them.

This will create a merge.shp file with a layer named “merge” containing all the data from other shapefiles.

PS: this works with shapefiles of the same type (eg. all polygon or all points).
PPS: this is mostly a reminder for me, since i will probably forget the merge command, and I dont want to lose any more time googling it.