Converting data from Microsoft Access (mdb) format to MySQL

Slovenian “National Interoperability Framework” (NIO) publishes some interesting data every now and then, but most data is usually in MS Access ‘mdb’ format and/or ‘csv’. Working with csv files is painful, since you have to do everything by hand (no selects, filters etc, you have to go through each and every line and check the conditions in a giant loop), and using Microsoft Access on Linux is an even bigger NO.

So, i decided to download some data (anonymized data on drivers licences in Slovenia), and convert it from mdb to MySQL (and insert it into the database). I could have done the same with parsing the csv file, but doing it from mdb autogenerates the database schema, (with csv, you have to do that yourself, by hand).

For converting the data i used mdbtools. The data i used can be found here (zipped mdb).

The first thing after downloading and unzipping the data, was to check which (and how many) tables are inside – you can do this with the “mdb-tables” command.

mulaz@x ~/vd $ mdb-tables Vozniki2013.mdb 

So, there is one table in the database named “Vozniki_2013” (old data, I know).

To generate the schema I used the “mdb-schema” command (with MySQL output):

mulaz@x ~/vd $ mdb-schema Vozniki2013.mdb mysql
-- ----------------------------------------------------------
-- MDB Tools - A library for reading MS Access database files
-- Copyright (C) 2000-2011 Brian Bruns and others.
-- Files in libmdb are licensed under LGPL and the utilities under
-- the GPL, see COPYING.LIB and COPYING files respectively.
-- Check out
-- ----------------------------------------------------------

-- That file uses encoding UTF-8

CREATE TABLE `Vozniki_2013`
        `VIR_PODATKOV`                  varchar (510), 
        `STAROST`                       varchar (510), 
        `SPOL`                  varchar (510), 
        `PODVRSTA_VD`                   varchar (510), 
        `DATUM_IZDAJE`                  varchar (510), 
        `DATUM_VELJAVNOSTI`                     varchar (510), 
        `OMEJITEV_220`                  varchar (510), 
        `OMEJITEV_70`                   varchar (510), 
        `OMEJITEV_71`                   varchar (510), 
        `AM_DATUM_PRIDOBITVE`                   varchar (510), 
        `AM_DATUM_VELJAVNOSTI`                  varchar (510), 
        `A1_DATUM_PRIDOBITVE`                   varchar (510), 
        `A1_DATUM_VELJAVNOSTI`                  varchar (510), 
        `AM2_DATUM_PRIDOBITVE`                  varchar (510), 
        `AM2_DATUM_VELJAVNOSTI`                 varchar (510), 
        `A_DATUM_PRIDOBITVE`                    varchar (510), 
        `A_DATUM_VELJAVNOSTI`                   varchar (510), 
        `B1_DATUM_PRIDOBITVE`                   varchar (510), 
        `B1_DATUM_VELJAVNOSTI`                  varchar (510), 
        `B_DATUM_PRIDOBITVE`                    varchar (510), 
        `B_DATUM_VELJAVNOSTI`                   varchar (510), 
        `C1_DATUM_PRIDOBITVE`                   varchar (510), 
        `C1_DATUM_VELJAVNOSTI`                  varchar (510), 
        `C1_95`                 varchar (510), 
        `C_DATUM_PRIDOBITVE`                    varchar (510), 
        `C_DATUM_VELJAVNOSTI`                   varchar (510), 
        `C_95`                  varchar (510), 
        `D1_DATUM_PRIDOBITVE`                   varchar (510), 
        `D1_DATUM_VELJAVNOSTI`                  varchar (510), 
        `D1_95`                 varchar (510), 
        `D_DATUM_PRIDOBITVE`                    varchar (510), 
        `D_DATUM_VELJAVNOSTI`                   varchar (510), 
        `D_95`                  varchar (510), 
        `BE_DATUM_PRIDOBITVE`                   varchar (510), 
        `BE_DATUM_VELJAVNOSTI`                  varchar (510), 
        `C1E_DATUM_PRIDOBITVE`                  varchar (510), 
        `C1E_DATUM_VELJAVNOSTI`                 varchar (510), 
        `C1E_95`                        varchar (510), 
        `CE_DATUM_PRIDOBITVE`                   varchar (510), 
        `CE_DATUM_VELJAVNOSTI`                  varchar (510), 
        `CE_95`                 varchar (510), 
        `D1E_DATUM_PRIDOBITVE`                  varchar (510), 
        `D1E_DATUM_VELJAVNOSTI`                 varchar (510), 
        `D1E_95`                        varchar (510), 
        `DE_DATUM_PRIDOBITVE`                   varchar (510), 
        `DE_DATUM_VELJAVNOSTI`                  varchar (510), 
        `DE_95`                 varchar (510), 
        `F_DATUM_PRIDOBITVE`                    varchar (510), 
        `F_DATUM_VELJAVNOSTI`                   varchar (510), 
        `G_DATUM_PRIDOBITVE`                    varchar (510), 
        `G_DATUM_VELJAVNOSTI`                   varchar (510), 
        `ID`                    int

Every field is “varchar (510)”, and that’s long. I’ve checked the data, and it fits into “varchar (50)”, so you can do a sed replace, to change the field size in the output. After that, you connect to the database, paste this output into the console (or pipe it directly from the command), and you will have an empty table created for your data.

This would also be a good time to create any indexes you need.

Now all we need is the actual data; we can get that using the “mdb-export” command (we have to tell it the table we want to export, and the output syntax).

mulaz@x ~/vd $ mdb-export -I mysql Vozniki2013.mdb Vozniki_2013
...many more lines (~1.3M)

All you have to do now, is to redirect this to a file, and copy it to te server with the database, and import it (or pipe it directly, if you can use the MySQL client on the machine you’re using now).

If you have a large table with alot of lines/INSERTs, you will notice that the import takes alot of time. The main slowdown for me was because MySQL commits the data after every INSERT statement, and that takes time. If you want to speed it up, you can disable the autocommit, and manually commit at the end. Since i’ve exported the data to a text file first, I just added the two lines in the beginning and the end of the file:

SET autocommit=0;
INSERT ....;
INSERT ....;

And voila, you have all the data in your MySQL database now.

Fixing video tearing on Intel Broadwell graphics

I have a Thinkpad X250 laptop (Core i7 5600U CPU) with integrated graphics. When I watch video, in some cases (usually window mode + flash player) causes very noticable video tearing effect (almost like interlacing). It usually didn’t bother me (I mostly watch video fullscreen, where there is no tearing).

And the fix? More obvious then i though. Create a file /etc/X11/xorg.conf.d/15-intel.conf with this inside:

 Section "Device"
   Identifier  "Intel Graphics"
   Driver      "intel"
   Option      "TearFree"    "true"

Yeah, who would have thought I had to enable the "TearFree" option…

Merging multiple ESRI SHP files into one

I’ve been playing around with GIS software for a couple of weeks now, and i’ve seen alot of datasources using tiled .shp files (instead of all the data in one huge .shp file, the data is split into “square” tiles). If you want to do stuff with that data, you sometimes want it in one .shp file, so here is a very simple script to merge the files to one:

for i in *shp; 
 do ogr2ogr -f 'ESRI Shapefile' -update -append merge.shp $i -nln merge; 

It just does a for loop over all shp files, and uses GDAL to merge them.

This will create a merge.shp file with a layer named “merge” containing all the data from other shapefiles.

PS: this works with shapefiles of the same type (eg. all polygon or all points).
PPS: this is mostly a reminder for me, since i will probably forget the merge command, and I dont want to lose any more time googling it.

gkrellm2 – changing the next/prev theme shortcut

Gkrellm2 is a “widget” that displays different system/etc. monitors at the side of the screen. It has keyboard shortcuts (PageUp and PageDown) to change the selected theme. So, if you change desktops and widow focus alot, it somethimes happens that gkrellm is focused by mistake, and by using the PageUp/Down keys, you change the theme when you didn’t want to.

I have no idea why, but the key combination is hardcoded in src/gui.c.

Here is the patch to change the combination to Control+PageUp:

diff -Naur gkrellm-2.3.5-old/src/gui.c gkrellm-2.3.5-new/src/gui.c
--- gkrellm-2.3.5-old/src/gui.c 2010-09-14 23:01:17.000000000 +0200
+++ gkrellm-2.3.5-new/src/gui.c 2016-07-20 11:31:48.256463308 +0200
@@ -2476,8 +2476,8 @@
 {"/-",                            NULL,        NULL,                                   0, "<Separator>"},
 {N_("/Configuration"), "F1",   create_config_window,   0, "<Item>"},
-{N_("/Theme/Prev"), "Page_Up", cb_load_theme,  THEME_ALT_PREV, "<Item>"},
-{N_("/Theme/Next"), "Page_Down", cb_load_theme,        THEME_ALT_NEXT, "<Item>"},
+{N_("/Theme/Prev"), "<control><shift>Page_Up", cb_load_theme,  THEME_ALT_PREV, "<Item>"},
+{N_("/Theme/Next"), "<control><shift>Page_Down", cb_load_theme,        THEME_ALT_NEXT, "<Item>"},
 {"/-",                            NULL,        NULL,                                   0, "<Separator>"},
 {N_("/Quit"),             NULL,        gtk_main_quit,                  0, "<Item>"},
 {"/-",                            NULL,        NULL,                                   0, "<Separator>"},


If you’re using gentoo. just put the file in /etc/portage/patches/app-admin/gkrellm and reemerge the package.

Aligator clips fail

Recently I’ve found a cheap USB boost converter (small device that turns a lower voltage (~1V-5V) source to a regulated 5V output – so you can charge your USB devices from a range of batteries with different (lower) voltages).

I’ve also received a pack of cables, with aligator clips on both ends, some time ago, so I’ve decided to use them instead of soldering/”improvising”.

Aligator clip cables
Aligator clip cables

For input power, I’ve used a TrustFire 18650 battery in a holder, and the aligator clip cables to connect the boost converter. On the output, I’ve also connected a USB power meter to measure the output voltage and current (I wasn’t sure if it was the 500mA or 1A module).

Boost module without load
Boost module without load

The voltage was 4.94 volts (somewhat low, but with such a small load, it could be the voltage regulator issue), and zero exit current. Everything is OK!

I’ve connected my powerbank to the USB port (to charge it, and check the current), and the situation got bad:

Boost module with load
Boost module with load

Current was around 120mA (well below useful), and the output voltage was 4.68V (also too low). So ok, it’s  a <$1 module, from eBay, probably no quality control whatsoever.. But just in case, let’s check the battery. I’ve connected a small 7segment LED display voltmeter to the boost module (under the clips), and nothing. The voltmeter requires ~3V to turn light up, so either it’s not working at all, or the voltage is too low. So I’ve used a proper multimeter, and measured the voltage of only 0.588 volts at the boost module! The batteries have protection circuits, so this isn’t a battery issue, since protection steps in at around 2.5 volts and shuts down the power completly. Voltage at the battery was around 3.8V when under load. Then I’ve touched the aligator clip cable with my hand (by accident), and it was warm to the touch – so here’s the problem!

Module voltage
Module voltage

Next thing, I’ve measured the voltage drop on the cables, and was surprised by the results (well, not really that much, since the cables heating up quite a bit). The voltage drop on the red cable was 1.657 volts and 1.378 volts on the black cable.  So, on a simple circuit with ~200mA of current, i was losing around 3 volts (~80% of the voltage) on the connecting cables.

Red lead voltage drop
Red lead voltage drop
Black lead voltage drop
Black lead voltage drop

What have I learned today? Always check/verify which cheap Chinese manufacturer to blame, so you don’t blame the wrong one!

Battery testing #1 – TrustFire 2400mAh

After setting up my battery capacity tester, I’ve decided to do a test on a TrustFire 18650 lithium battery I’ve bought ~1.5 year ago on DealExtreme (url). The battery specs (and label) say it’s a 2400mAh battery, but with cheap chinese batteries, it’s better to measure then believe!

Trustfire 2400mAh battery
Trustfire 2400mAh battery

The battery i have is not new, but has seen very little use (probably about 4-5 charge/discharge cycles), and has been fully charged (not good, I know) in a small battery box at room temperature for most of that time, so i was expecting some loss in capacity.

I’ve charged the battery to full using a XTAR MP15 battery charger (USB power, very slow, but seems to work well), and started the discharge measurement immediately after the LED on the charger turned green (the battery should be at 95%+ capacity then, and the charger stays in CV mode for some time after that).

And the results? With a cutoff voltage of 3.0 volts (the protection circuit should kick in at about 2.5 volts, but I wasn’t willing to risk overdischarging it), the measured battery capacity was 2.271mAh! Which is great for that price!

So now, on my TODO list: testing AA/AAA batteries, brand name vs. much cheaper, store branded ones.

Another cheap gadget: battery capacity meter

Recently, I’ve been browsing ebay for random electronics, and I’ve found a cheap ($4.33) battery capacity meter. Since it was below my $5 “I wont impulse-buy it, I’ll think about it”, i immediately ordered it.

It came today, packed in bubble wrap, and contained a (micro) USB powered meter and a 5W 7.5Ohm load/resistor (measured 7.8Ohm at room temperature). On the left side are four terminals, outer two for the resistor and inner two for connecting the battery. In the middle is 4-digit 7-segment display, cutoff voltage adjustment buttons and three display leds, and on the right side is a micro USB connector, suppling power to the circuit. The microcontroller is probably hidden under the LED display.

The meter is rated for 1.5-12V with a maximum load of 3.1A, so I’ve decided to try it out with a 18650 battery which should power a current of about 0.5A through the provided load. I’v also connected the load and the USB power connector. After powering on it showed the voltage of about 3.79V on the LED display.

Voltage display
Voltage display

Using the (+) and (-) keys, you can adjust the cutoff voltage – this is the voltage when the battery is considered “empty”. Usually, most lithium cells have extra protection circuits, which cut off the power, when the cell voltage is too low, to prevent over-discharging – so you have to consider that when setting the voltage limit.

Setting cutoff voltage
Setting cutoff voltage

Pressing OK starts the discharge process. The display loops between capacity (Ah), current (A) and voltage (V), and shows the current values.

Display while discharging
Display while discharging

I’ve verified the values with my multimeter – the voltage was the same on both meters in all three digits, and the current differed only slightly with the last digit (<1mA difference compared to my multimeter).

Warning:  the resistor gets HOT (>100°C). I’ll probably replace it with something larger or add some heatsinks to it.

I’ve tweaked the cutoff voltage (so I could see what happens when it’s finished measuring), and the display started flashing rapidly and stayed in the capacity display mode.

Finished measuring
Finished measuring

The meter also shows a few error codes, if you mess something up (set up cutoff voltage below the current battery voltage, etc.):

Err1: the battery voltage higher than 15V
Err2: the battery voltage is lower than the stop voltage
Err3: the battery is unable to withstand the load discharge current
Err4: the current is too large (current is more than 3.1A)

Considering the price, the accuracy (within reason of course, my multimeter hasn’t been calibrated in sime time too), and the overinflation of battery capacities in the specifications (eBay sellers, I’m looking at you!), I consider it a nice gadget to have, to test your purchases, before relying on the written spec (eg. “20000mAh” 18650 cell)

Copying a file increasing in size

This is a quick hack, for when I used udpxrec (part of udpxy) to record an IPTV stream to an mpeg file, but instead of saving it to a network share (to watch it with some delay on my OpenElec box), i saved it to a local drive (by mistake). So here was a file, gradually increasing in size, which i wanted on my network drive (to start watching before the actual show/recording is finished). cp of course wont work, since it stops when it detects the end of file (does not detect new data being added, and wait for it), so you need to use something else.

When you think of a file with data being appended at the end, the first thought is “tail -f” (-f = follow and print the data being appended). Since tail only prints the last few lines (or bytes), you need to set it to output from the beginning with the “-c +0” (output bytes starting at the zero-th byte). I also pipe it through pv to follow the progress and copy rate.

So the command is:

tail -f -c +0 /path/source.mpg | pv > /destinationpath/destination.mpg


Playing with a cheap ($10) logic analyzer

A long time ago I ordered (eBay) and recieved a very cheap logic analyzer (it came out to be a Saleae clone). I havent really played with it enough, but I decided to test it.

It connects to PC via mini(!)-USB cable, and has 8 (+2 gnd) pins on the opposite end. The label marks it as a 24MHz 8 channel analyzer. I’ve connected it to a USB->RS232 adapter (from an old mini/nano Arduino kit).

Logic connection

I expected it needed some weird Chinese software, but it worked with original software from Saleae (which is nice).

Screenshot with data already captured

Capturing data is also very easy – the only two options are sample rate and data capture duration.

Data capture options
Data capture options

After the capture timer is over, the program shows logical levels on a time graph (first screenshot) – it does not seem to autodetect the protocol used. After selecting the protocol and protocol options (mainly the correct pin/s) it also decodes the actual data (“test123” in this case).

Supported protocol decoders
Supported protocol decoders
Async serial options
Async serial options

As it can be seen on the first screenshot, it correctly decoded the data sent via the RS232 adapter.

For $10, it’s a great toy. It probably has some timing/sync issues, and the casing isn’t that great, but for that little money, who cares!